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FOREWORD 



The general purpose of bilingual vocational training (BVT) is to provide vocational 
training and other employment-related services (including English-language 
instructloi) to limited English proficient (LEP) adults. IThile some BVT projects 
have been able to document the extent of their success , others have not focused on 
program evaluation as an important priority. Since e\raluation results can be 
extremely helpful for improving program planning, implementation, and management, 
this Guidebook has been prepared to help project directors and evaluators with 
guidelines for evaluating their local projects. 

Program evaluation can be a important tool for project managers. It can be used to 
assess whether a project is being implemented as planned, and to assess the extent 
to which Btated goals and objectives are being achieved. Thus, evaluation data 
provide the basis for planning and dec is ion "-making relative to program improvement. 

In addition to being extremely useful for local project improvement, evaluation 
results are important for program improvement and policy-making at federal and 
state agencies. However, to be maximally useful at those levels, data must be 
standardized so that they can be aggregated across local projects. Therefore, a 
reporting form has been developed for collecting consistent program, participant, 
and impact data. This report is discussed in Appendix A and directions for docu- 
menting English language proficiency in the report are discussed in Appendix B* 
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1. INTRODUCTION 



The BVT Model 

Bilingual vocational training (BVT) is a program designed to assist limited 
English proficient (LEP) adults in need of job skills, employment and related 
English language proficiency* In addition to having communication difficulties 
associated with a lack of English language skills, LEP adults are g(2inerally 
unemployed or underemployed, and have substantial needs for Job counseling and 
placement. A BVT program uses the native language of the trainee for 
developing vocational skills and teaching related vocational subject m;^.tter, 
while developing the trainee *s ability to use and understand English needed for 
the job. 

More specifically, the "BVT Model" is composed of the following activities 2 

1. Recruitment: activities designed to specifically 

attract LKP adults; 



2. Intake and Assessment: 

3. Bilingual Vocational Instruction: 

4. Vocational English as a 
Second Language vVESL): 

5. Counseling and Support Services: 

6. Job Development and Placement: 

7. Coordination: 



activities for gathering information from 
participants to describe their general 
characterlatics, vocational interests and 
aptitudes, and English language 
proficiency; 

instruction which uises a combination of 
English and the trainees* native language 
to teach vocational skills; 

instruction that focuses on job-specific 
and employnnent •-related English; 

individual assistance In employment, and 
related ad;^nstment problems; 

outreach with local employers to place 
program participants In wage-earning jobs 
after training; and 

activities to link the other six 
components so that they mutually support 
each others 
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BVT programa have four fundamental goals s 

• Stable employment for limited -English proficlevit (LEP) adults; 

• Increased Job-related English language proficiency for LEP adults; 

• Increased earning capability for L'EP adults; and 

« Greater carecir advanceraant opportunities for LEP adults, 

Tlie evaluation strategy presetited in this guidebook U directed at aaseaalng 
the extent to which these goals have been achieved* 

^* Pwfposes of 'This Evaluatla n Guide 

The overall goal of this Guide is to help individual projectb improve, their 
approach to program evaluation. At the local project level, there is a need to 
obtain more comprehensive data in order to assess effectiveness and improve 
program planning and management. The Guide presftnta an overall framework which 
can be used by local BVT project staff and ssvaiuators to help establiah the 
scope of their evaluations and to give direction to projects in meeting local 
evaluation needs. There la also a need for standardized data which can be 
aggregated across a group of projects to present an overall picture of a State 
or Federal funding program. The standardized data can then be used for 
planning, policy, and management purposes* The?, BVT Statistical Summary Report 
(Appendix A) provides a mechanism for this latter purpose. 



What the G uide Is Not Iixtended to Accomplish? 

In order to be as clear as possible about the purposes and intended uses of 
this Guide, a word la In order about what It Is not intended to do for the 
various users. It is not intended to be a "textbook" about evaluation in 
general, nor about evaluation of vocational programs in particular. 



Specifically, it la TWjt intended tos 



• be a "cookbook" for evaluating local programs with specific recipes and 
menus for each aspect of an evaluation; 

« be considered a lock-step methodology with a compendium of precise 

Instruments for examining all of the activities and services of the BVT 
Model; nor 

• be a limiting or reatrlctlvu directive^jan what should or should not be 
evaluated in each local program. O 
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Since the features and needs of different projects vary greatly ^ specific 
measures and procedures for assessing vocational skills are not goalvS of this 
Guide. Each program may adapt the materials in this Guide to its own needs, 

ffliat the Guide la I ntended to Ho 

/ 

In the broadest sense > the Guide will provide guidance and assist local 
programs to design and carry out their local evaluations* Thus, the Guide Is 
intended to: 

• provide direction for planning a relevant evaluation which meets the 
unique needs of each local program? 

• provide a cciapendium of evaluation questioas around which a local 
evaluation can be structured; 

• provide sample instruments for collecting data at program intake j at 
program completion, and at follow-up periods after job placement; and 

• provide specific guidance for carrying out critical data collection 
activities, such as measuring English proficiency and collecting 
follow-up data from former participants and from employers • 

C. Audiences 

The primary audlenca for this Guide consists of local BVT project directors, 
staff, and evaluators. Others who may profit from using the Guide are state 
program administrators and coordinators who wish to know more ibout the impact 
of local programs across their States. State special needs coordinators and 
state vocational education directors can use the Guide for evaluating their 
local projects. 

Another group which can use this Guide includes the Federal BVE Program staff 
and other officials Interested in national level data concerning the Impact of 
vocational training for LEP adults. The BVT Statistical Summary Report (see 
Appendix A) can be used to create a national data base concernlre the effects 
of participation in BVT programs. The results can be used to plan future 
programs, set program policy, allocate resources, promote the program 
nationally, and manage/monitor the programs in general.* 
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In surainary, this document is called a Guide for dciliberate reasons. The local 
program staff and evaluator will be able to use che framework, the evaluation 
questions, sample instruments, and Information on English language proficiency 
measurement, follow-tip procedures, and other aspects of data collection to fit 
the needs and Issues which concern their local program. The listing of 
evaluation questions and the sample instruments presented in this document are 
not Intended to be limiting by any means. They provide a starting point to 
facilitate the process of formulating evaluation questions and focusing the 
evaluation. 

AAA 



The next chapter of this Guide presents a six-step evaluation process. This 
process is general in nature and discusses how an evaluation should be planned 
and conducted. Chapter III, on '.he other hand, sets forth a framework for 
specifically evaluating local BVT projects. This framework discusses what 
aspects of the BVT program should be included in the evaluation, what questions 
should be addressed, and what data should be collected. The general process 
described in Chapter II should be applied to the specific framework presented 
in Chapter III in order to plan and conduct a comprehensive evaluation. 
Chapter IV - VII of the Guide provide guidance in planning for each component 
of the evaluation framework, while Chapter VIII discusses how the data and the 
evaluation findings may be presented. 
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II • EVALUATION PROCESS AND PLANS 

Evaluation is one of the basic tools for program management and Improvement. The 
central questions to be answered Include: 

^ To what extent have program objectives been met? 

• What contributed to the successes and failures? 

• ^^at changes and Improvements should be made? 

Thus, evaluation provides information for documenting and improving program 
effectiveness. 

The documentation of program effectiveness involves the systematic collection, 
analysis, and reporting of information or data. It should not be forgotten, 
however, that the Improvement of program effectiveness involves human judgement, 
I.e., using tl.a reported data for planning and decision-making which will result in 
program Improvement and Increased effectiveness. 

A« Overview of the Evaluation Process 

The evaluation process can be described as Involving six progressive steps. 
Ttiese steps are shown in Exhibit 1, and are discussed below. It is important 
to remember that initiating an evaluation cannot wait until a program is 
nearlng completion. An evaluation should be planned as part of the overall 
program and should be Implemented at the same time the program begins 
operation. In this manner, program activities can be adequately documented 
from their beginning, and baseline data on program participants can be 
collected as they enter the program. 

Step 1 : Defining the Purpose and Scope of the Evaluation 

The first step in planning an evaluation is to define its purpose and 
scope. This helps set the limits of the evaluation, confining it to a 
manageable size. Defining its purpose includes deciding on the goals 
and objectives for the evaluation, and on the audience who will use 
the evaluation results. The evaluation goals and objectives may vary 
depending on whether the program being evaluated is new and is going 
through a try-out period, or if a program has been thoroughly tested 
and needs documentation of its success before information about the 
program is widely disseminated and adoption by others encouraged. 



ERIC 



DEVELOPMENT ASSOCIATES. INO. 



6- 



EXHIBIT 1 



STEPS IN THE EVALUATION PROCESS 



1. Define the 

Purpose and Scope 
of the Evaluation 



2. Specify the Evaluation 
Questions 



3. Develop the Evaluation Design and 
Data Collection Plan 



4. Collect the Data 



5, Analyze the Data 

and Prepare the Report 



6. Use the Evaluation 
Report for Program 
Improvement 
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Depending on the purpose, the audience for evaluation may be 
restricted t' the project director and his/her staff, or may include a 
wider range of individuals, from agency administrators and 
decision-makers, to planners and other officials at the local, state, 
or federal level. 

The scope of the evaluation depends on the evaluation's purpose and 
the information needs of its Intended audience. These needs detertaine 
the specific components of a program which should be evaluated and on 
the specific project objectives which are to be addressed. If an 
evaluation of a program has recently been conducted, a limited 
evaluation may be designed to target on certain parts of the program 
which have been changed, revised, or modified. Similarly, the 
evaluation may be designed to focus on certain participant objectives 
which were shown to be only partially achieved in a previous 
evaluation. Thus, this step will define exactly which aspects of the 
program are to be evaluated. Costs and resources available to conduct 
the evaluation must also be considered in this decision. 

Step 2 : Specifying the Evaluation Questions 

Evaluation questions are general questions that grow out of the 
purpose and scope specified in the previous step. They help further 
define the limits of the evaluation. The evaluation questions are to 
be discussed and answered in the evaluation report and should be 
formulated to address the needs of the specific audience to whom the 
evaluation Is directed. Evaluation questions should be developed for 
each component of the program which falls into the scope which v;as 
defined in the previous step. For example, questions may be 
formulated which concern the adequacy of the curriculum and the 
experience of the teaching staff; other questions may concern the 
appropriateness of the vocational skills being taught in relation to 
employment opportunities in the local community; additional questions 
may relate to the appropriateness of the individuals being recruited 
for the program with respect to their English language proficiency and 
etnployment status; and finally, evaluation questions may relate to the 
extent to which participants are achieving the goals of the program, 
such as stable employment and increased earning capability. 

A good way to begin formulating evaluation questions is to carefully 
examine the project's objectives; another source of questions Is to 
anticipate problem areas concerning program implementation. 
Importantly, the audience for or expected users of the evaluation 
should be involved in Jeveloping the evaluation questions. This 
should never be left solely to the outside evaluator, no matter hov^' 
familiar he or she is with the program. Once the evaluation questions 
are developed, they should be prioritized and examined in relation to 
the time and resources available. Once this is accomplished, the 
final set of evaluation questions can be selected. 
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Step 3 : Developing the Evaluation Design and Data Collection Plan 

This step Involves specifying the approach to answering the (^.valuation 
questions, Including how the required data will be collected* This 
will Involve: 

- specifying the data sources for each evaluation question; 

- specifying the types of data collection approaches and Instruments 
needed ; 

- specifying the specific time periods for collecting the data; 

- specifying the staff members who will be assigned to collecting the 
data, and how their data collection responsibilities relate to 
their other project responsibilities; and 

- specifying the resources which will be required to carry out the 
evaluation. 

The design and data collection plan is actually a roadmap for carrying 
out the evaluation. An important part of the design is the 
development or selection of the instruments for collecting and 
recording the data needed to answer the evaluation questions « Data 
collection Instuments may vary from record -keeping forms, 
questionnaires, interview guides, to vocational and language skills 
tests. Some of the instrumentation may already be available, l^e*, 
forms used for recordkeeping and management purposes, such as 
rr sruitraent or intake forms, etc. Some of these forms will have to be 
modified to meet the evaluation needs. In other cases, new 
instruments will have to be created. 

In designing the instruments, the relevance of the Items to tue 
evaluation questions and the ease or difficulty of obtaining the: 
desireri data should be considered* Thus, the instruments should be 
reviewed by the project director and staff members to ensure that the 
data can be obtained in a cost-effective manner and without causing 
major disruptions or inconveniences to the project. 

St^p 4 ; Collecting the Data 

Data collection should follow the plans developed in the previous 
step. The individuals assigned to the various data collection tasks 
need to be thoroughly trained in the data collection requirements and 
procedures. Only by following standardized procedures will the data 
be reliable and valid. Following training, the project director needs 
to monitor the staff to ensure that they are accomplishing their data 
collection assignments according to the specified time schedule. The 
data should be recorded carefully and neatly so they can be read and 
interpreted during the analysis stage. Proper record -keeping and 
filing are similarly important so that the data are not lost or 
misplaced. Any problem should be discussed with the project director 

14 
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and evaluator* Deviations from the data collection plan should be 
documented so that they can be considered In analyzing and 
Interpreting the data. 

Step 5 : Analyzing the Data and Preparing a Repor t 

This step Involves tabulating, summarizing, and Interpreting the 
collected data In such a way as to answer the evaluation questions. 
These procedures should be compatable with the type and amount of data 
which were collected, and the goals and objectives of the evaluation. 
Appropriate descriptive measures (frequency and percentage 
distributions, central tendency and variability, correlation, etc.) 
and inferential techniques (significance of difference between means 
and other statistics, analysis of variance, ch-square, etc.) should be 
used to analyze the data. The local evaluator should have 
responsibility for this aspect of the evaluation. 

The evaluation will not be completed until a report has been written 
. and the results communicated to the project director and other 
appropriate administrators and decision-makers. In preparing the 
report, the writers sb>>uld be clear about the audience for whom the 
report is being prepared. Two broad questions need to be considered: 
(1) What does the the audience need to know about the evaluation 
results? and (2) How can these results be best presented? Different 
audiences need different levels of information. Administrators need 
general information for policy decision-making, while project staff 
may need more detailed information which focuses on project activities 
and effects on participants. 

The report should cover the following: 

- The goals of the evaluation; 

- The procedures or methods used; 

- The findings} and 

- The implication of the findings including recommendations for 
changes or improvements in the program. 

Importantly, the report should be organized so that It addresses all 
of the evaluation questions specified in Step 2. 

Step 6 : Using the Evaluation Report for urogram Improvement 

The evaluation should not be considered successful until its results 
are used by program managers and decision-makers for program 
improvement. After all, this is the ultimate reason for conducting 
the evaluation. The evaluation may indicate that a program activity 
is not being implemented according to plan, or it may indicate that a 
particular objective is not being met. If this does occur, ^.t is then 
the responsibility of the project director to make appropriate changes 
to remedy the situation. Project directors should never be satisfied 
with their programs. Improvements can always be made, and evaluation 
is an important tool for accomplishing this purpose. 

15 
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B. Planning the Evaluation 

ft 

The evaluation should be conducted by an independent, experienced evaluator. 
This individual will provide the expertise for an evaluation which is 
comprehensive, objective, and technically sound. The project director and 
her/his staff must work closely with the evaluator beginning with the planning 
stage to ensure the evaluation meets the exact needs of the project. 

Adequate time and thought for planning an evaluation Is essencial, and will 
give the project director and staff an opportunity to develop ideas about what 
they would like the evaluation to accomplish. The evaluation should address 
the goals specified in the project application and management plan. In some 
projects, however, one or more goals or objectives may require special 
attention. Some activities or instructional strategies may have been recently 
implemented, or the staff may be aware of some special problems which should be 
addressed. For example, there might have been a recent break<^c<m in 
communication between the ESL teachers and the vocational instructional staff ; 
or the characteristics of the participants in recent training cycles mi,<?ht have 
begun to differ significantly from past groups, having implications for 
vocational training or the approach to language instruction. These are 
examples of things which should be considered when the project director selects 
an evaluator. The evaluator must then familiarize himself oi* herself with the 
special issues of concern on which the evaluation should focus. 

Thus, the initial step of the evaluation process iw;'"''"^9.s thinking about any 
special needs which will help in planning the overall evaluation and selecting 
I'le evaluator. Special evaluation questions and problems identified in the 
instructional staff area might sugg'^st that an evaluator is needed who has 
expertise in evaluating instructional systems, etc. Similarly, If the project 
needs an evaluation which requires frequent on-site observation of teaching 
methodologies by the evaluator, then this will help the project director focus 
on hiring someone located nearby so that travel costs can be kept to a minimum. 
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In nummary, defining the scope involves setting limits, identifying specific 
areas of inquiry and deciding on what parts of the program and on which 
objectives the evaluation will focus • The ncope does not answer the question 
of how the evaluation will be conducted. In establishing the scope, one is 
actually determining which components or parts of the program will be 
evaluated. This step is important and, indeed, implies that the evaluation may 
not cover every aspect and activity of the program. 

S electiHo the Evaluator 

Selecting an evaluator for the program is one of the most important elements in 
ensuring a technically sound and useful evaluation. The basic criteria 
suggested for selection are as follows: 

• Skills in evaluation design, constructing data collection instruments, 
collecting data, managing and maintaining quality control over data 
collection, analyzing data, and writing reports; 

• Experience in conducting evaluations of vocational training projects 
targeted at LEP adults; 

• Knowledge of the BVT model; 

• Experience in collecting data from employers and community and business 
groups ; 

• Ability and willingness to work directly with the project director in 
order to design the evaluation, oversee its implementation, and prepare a 
report ; 

• Reside within reasonable distance of the project so that travel costs are 
minimal and scheduling work sessions is not a problem; and 

• Available for complete time period required for the project at a rate 
that fits the basic budget resources allocated to evaluation and the 
number of evaluator-^ays projected in the initial planning; the project 
might expect to plan for at least 20-25 work--days for the evaluator. 

Once an individual has been selected and has agreed to become the local 
evaluator for the project, a contract and work plan must be developed so that 
expectations, roles, and responsibilities are clear to all parties. 
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D. S pecifying the Roles of Prrject Director. Staff and Evaluator 

In order for the evaluation to be planned and carried out effectively, the 
roles of the project director, staff, and evaluator must be made clear to all 
parties. The evaluator should be respons''le for specifying the design and 
developing the data collection Instruments, training project staff to collect 
the required data, analyzing the data, and preparing a comprehensive report. 
The project director should work with the evaluator in specifying the 
objectives and scope of the evaluation, and assigning and supervising the 
project staff In carrying out their assigned data collection and record-keeping 
tasks. The project staff should be assigned the major data collection and 
record-keeping tasks for the evaluation. This will Increase their job 
responsibilities, but is the most cost-effective way of collecting the required 
data. More specifically, the suggested roles of the project director, the 
evaluator, and the project staff are listed below: 

The project director will: 

• recruit and hire an experienced evaluator; 

• work with the evaluator to define the objectives and scope of the 
evaluation; 

• work with the evaluator to define the evaluation questions which will be 
addressed; 

• review and approve data collection Instruments and procedures to ensure 
that they are compatible with project activities; 

• assign and supervise project staff for data collection and recordkeeping; 

• approve schedule and expenditures for the evaluation; and 

• approve final evaluation report. 
The evaluator will; 

• work with the prjject director to define the purpose and scope of the 
evaluation; 

• work with the project director to specify the evaluation questions which 
will be addressed; 

IS 
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t develop the evaluation design; 

t develop data collection instruments and procedures, including the 
schedule and calendar for all data collection; 

t train project staff to collect the required da^a; 

t analyze the collected data to answer the evaluation questions; and 

• prepare the evaluation report. 
The project staff will: 

t collect the evaluation data; 

• maintain participant tracking records as well as conduct follow-up 
surveys of former trainees; and 

t assist the evaluator in tabulating and summarizing the collected data* 
E. Allocating Resources 

Critical questions in planning the evaluation concern how much staff time and 
financial resources can be expended on the evaluation 9 and what resources are 
needed to actually carry it out. These two questions need to be addressed by 
the project director and the evaluator early in the design process. Many 
evaluators will raise these questions in the initial exploratory discussions 
concerning the evaluation. Preliminary planning will have to be accomplished 
before a final decision can be made on both the role and level of effort of the 
evaluator and each project staff member. 

A variety of design decisions must be made during the planning stages that 
affect the allocation of both financial and staff resources. Each decision 
affects the staff /evaluator roles and functions, level of effort, and 
resources, and ultimately determines the overall scope of the evaluation. For 
Instance, a record-keeping procedure must be developed to maintain accurate 
evaluation data. A tracking system Is also needed to manage follow-up surveys 
of former trainees. These record -keeping and tracking systems, as well aa the 
actual data collection activities, involve staff and material costs. These 
costs need to be considered in defining the final scope of the evaluation. 
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To summarize, this chapter presented a general process which focused on how an 
evaluation may be planned and carried out^ Ttie next chapter, and in fact the 
remainder of this Guide, dicusses vrtiat program components, evaluation 
questions, and data should be included in an evaluation of a local BVT project. 
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III. EVALUATION FRAMEWORK 



This chapter presents a framework for evaluating local BVT projects which combines 
outcome evaluation with process evaluation. An outcome evaluation attempts to 
determine the extent to which a project's specific objectives have been achieved. 
On the other hand, the process evaluation seeks to describe the program which was 
implemented, and through this, attempt to gain an understanding of why the 
objectives were or were not ajhieved. 



Evaluators have been criticized in the past for focusing on outcome evaluation and 
excluding the process side, or focusing on process evaluation without examining 
outcomes. The framework presented here incorporates both the process and :)utcome 
side. In this manner, one can determine the effect (or outcome) of a program, and 
also understand how the program produced that effect and how the program might be 
modified to produce that effect more completely and efficiently • 

In order to focus on both process and outcomes of a local BVT program, an 
evaluation should be designed in which evaluation questions, and data collection 
and analysis, address the following: 



• Program Environment; 

• Program Participants; 

• Program Implementation; and 

• Program Outcomes. 

These components may be thought of as being linked in the folloving manner; 



Program 
Environment 



Program 




Participants 





Program Implementation 



Program Outcomes 
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Using this framework, descrlptlona avj prepared of Khe. environment, the 
participants, and the program activities and services which are implemented. 
Outcomes of the program are also assessed. The description of the environment, 
participants, and activities and services are used to explain how the outcomes were 
achieved and to suggest changes which may produce these outcomes more effectively 
and efficiently. 



Each evaluatlou component is described below. 



Environment 

This component defines the environment in which a BVT project is implemented. 
This includes the community In which the project Is located, the economy of the 
community, the job opportunities in the, community, and the characteristics and 
job training needs of the LEP adults who live In the community. Understanding 
the environment in which the project is located is Important to assessing and 
interpreting the outcomes of the program, especially job placement, job 
retention, and Job advancement. Data concerning this component are collected 
through a project needs assessment which should be completed prior to funding. 
The data are generally presented in the project grant application. 

Participants 



This component defines the characteristics of the participants. Including 
English proficiency at program entry, previous vocational training, and work 
history. Data on the characteristics of participants may be collected as part 
of program recrultuient and intake activities. In addition to. their use for 
descriptive purposes, these data are useful for comparislons with similar data 
collected at completion of the training and at follow-up data collection 
periods. 

Program Implementation 



This component describes how the key activities of the program are implemented, 
including recruitment, intake and assessment; vocational and English language 
instruction; counseling and other support services; and job development and job 
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placement* In this manner, the outcomes or results achieved by the program can 
be attributed to what actually has taken place, rat.*er than what was planned to 
occur • This component also addresses the questions of what services and 
activities of the program have been fully implemented,, partially implemented, 
and not implemented ♦ 

Outcomes 

This component concerns the effects that the program has on its participants, 
and to what extent the program has met its stated objectives for program 
participants. Data concerning these objectives should be collected at program 
completion and at specific times following the initial job placement of each 
completer. 

At program completion, data should be collected on rates of program completion, 
achievement of vocational and Engliah language instructional objectives and 
competencies, vocational skills and knowledge, English language proficiency, 
placement in a job, wages per hour, hours worked per week, and satisfaction 
with the training received. 

At both 90 and 180-days following Initial job placement after training, data 
should be collected on employment rates of former participants, wages per hour, 
number of hours worked per week, number receiving job promotions, number 
receiving wage increases, ratings of job satisfaction, and ratings of employer 
satisfaction with the vocational and English language skills of employees v/ho 
had completed the BVT program. 

Using the above four evaluation components, a comprehensive assessment of a BVT 
program may be designed. Not only will this evaluation approach allow the project 
staff to determine the extent to which project goals and objectives are met, but 
will also onable them to understand b those outcomes were achieved and to make 
. changes for program improvement purposes in the future. 
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The evaluation framework presented above should be Implemented using the six-step 
process described In Chapter II. The framework describes what should be included 
in the evaluation; the six-step process describes how the evaluation is to be 
planned and carried out. Guidelines for defining the scope of the evaluation, 
specifying evaluation questions, and developing the data collection plans for each 
of the four evaluation components are discussed in the following chapters. 
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IV. PROGRAM ENVIRONMENT 



This chapter focuses on obtaining descriptive information concerning the economic 
and employment environment of the community in which the project Is located. Thus, 
it provides a context against which job placement, wages, and other outcomes may be 
measured . 

Information on the economic and eaployment characteristics will most often be 
collected as part of a needs assessment which is generally conducted prior to 
planning a project. In fact, the needs assessment results are often part of the 
project's grant proposal or application. Thus, no new data may be required; the 
needs assessment data may satisfy the requirements of the evaluation. 

In focusing on the desired information for this aspect of the evaluation, the 
project director and evaluator should decide on the purposes that information on 
the environment of the program may play in the overall evaluation plan. This will 
define the scope of this part of the evaluation. From this, the relevant 
evaluation questions may be developed. 

A set of questions which relates to the program environment is provided in Exhibit 
2. These ire illustrative questions. The final set of questions should be defined 
and developed by the project director and evaluator so that they meet the needs of 
their local program. 

After the questions are specified, the evaluator should identify the specific 
variables which are applicable to the questions, and the possible sources of data 
which may be used to answer the questions. Examples of variables and data sources 
are shown in Exhibit 2. Following this, a strategy for data collection and the 
data collection instruments themselves needs to be developed. However, much of the 
data should be available from the project grant proposal or application. 
Therefore, new data may not have to.be collected unless the existing data are 
outdated. If new data are required, they may be available from community agencies 
such as the local economic development agency, local or state employment 
department, and Chamber of Commerce. 
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The findings which result from this part of the evaluation will eventually help 
explain the results of the outcome evaluation. For L;xample, if project goals 
concerning placement and stable employment are not achieved, it may be because the 
project is providing training in vocational areas which are inappropriate for the 
present economic environment of the local community (no jobs are available)* Thus, 
the evaluation results can help the project director make programming decisions to 
rectify the situation. 
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EXHIBIT 2 

PR0G3EIAM ENVIRONMENT: EXAMPLES OF EVALUATION 
QUESTIONS ^ VARIABLES, AND DATA SOURCES 



Evaluation Questions 



Variables 



Possible Data 
Sources 



1* ^That are the general character- 
istics of the community served 
by the BVT project? 



2. What are the employment and 
economic characteristics of the 
community in which the project 
is located? 

3* How many LEP adults in the 
community need employment 
and related services? 



4« How many LEP adults need 
vocational training? 

5* T^hat kinds of jobs are avail- 
able in the community for which 
the BVT project could train 
LEP adults? 



Total population; number 
of LEP adults by language 
group; demographics of the 
LEP adults. 

Unemployment rate; mean 
family Income of general 
population and LEP popu- 
lation 

Number of LEP adults 
seeking employment; 
number of LEP adults 
unemployed; number of LEP 
adults underemployed. 

Number of LEP adults 
needing BVT 

Jobs available in 
community 



Published 
Census Reports 



Local Labor 
Department 



Local Labor 
Department 



Local Labor 
Department 

Local Labor 
Department 



6. Which areas of vocational 
training are most needed 
by the LEP adults in the 
community? 



Typejj of jobs avail- 
able in community 



Local Labor 
Department 
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V. PROGRAM PARTICIPANTS 

This chapter concerns tb 't part of the evaluation related to the number and 
characteristics of the participants being trained by a BVT project. These data 
will be collected during recruitment periods or upon entry to the program. Most 
participant data will be descriptive In nature, such as age, ethnicity, education, 
years lived In U.S., etc* Some data, however, will be baseline measures related to 
project objectives. These data will be compared to data collected at completion of 
training to determine project effects. Es:arople8 of these data are language 
proficiency scores, employment status, and wages at or prior to program entry* 

The specific data to be collected on participants should be determined by the 
project director and evaluator, and depends on the Issues which they choose to 
address* From this, a set of evaluation questions should be developed by the 
evaluator which focus on these Issues. This then leads to specification of the 
variables on which data should be collected, and the development of the data 
collection plans and Instruments* 

Evaluation questions which concern program participants are shown In Exhibit 3 
along with examples of the relevant variables' and data sources. These questions , 
are examples, not all of which may be of Interest to a particular local program. 
Conversely, a particular local program may wish to add questions* 

A project recruitment or Intake form will generally supply most. If not all, of the 
information needs concerning the characteristics of project participants. The 
evaluator should examine the existing form(s) used by the project to determine 
whether all data needs are satisfied. If not, these forms should be modified so 
that the additional required information is collected* 

A sample intake form is shown in Exhibit 4. The form can be adapted to meet the 
needs of any local program. Thus, additions, deletions, and modifications to the 
form may be made as desirable. The staff members responsible for recruitment and 
Intake will then record the required intake data for each particip^-.t on these 
forms. The evaluator should then be responsible for summarizing the data across 
participants* 
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Except for the measurement of language proficiency, the data required by the 
questions in Exhibit 3 and the sample Intake form In Exhibit 4 can be collected 
through Interviews with program participants during recruitment or intake* 
Language proficiency measurement » discussed below, is more complicated. 

The measurement of participants* English language proficiency is necessary for two 
reasons. First, it will provide the data for describing how well individual 
participants are progressing in increasing their English language skills. Second, 
It will provide the basis for assessing how well the project as a whole has met the 
goal of increasing job-related English language proficiency. 

In order to select an appropriate language proficiency instrument. It Is Important 
to consider specific project goals and needs, as well as the characteristics of the 
Instrument Itself. Thus, the adequacy of any language proficiency measure should 
be assessed according to the criteria below: 

• Is the instrument appropriate for adults ? 

• Does the measure assess the appropriate language areas? 

• Is the meas^^rement appropriate, i.e., does it take a discrete-point or 
integrative approach? 

• Is the Instrument technically sound, i.e., is the validity and reliability 
of the Instrument adequate? 

The first criterion requires project staff to assess the appropriateness of the 
Instrument for limited English proficient (LEP) adults . This step is extremely 
Important since there may be the temptation to simply adopt an existing Instrument 
designed for LEP school-age youth. In rare Instances, such an Instrument could be 
used. However, in the majority of cases, such use would be inappropriate and the 
resulting information would be Inadequate. If an Instrument is not designed for an 
adult population, it will be critical to evaluate the content of the measures to 
make certain that it is appropriate for adults. 

The second criterion requires that project staff select an Instrument which 
measures the language skill areas which need to be assessed. The areas that can 
potentially be measured Include all components of listening, reading, speaking and 
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writing. Before an instrument is selected, the project staff need to decide on the 
specific skill areas to be measured. This judgment should be made on the basis of 
project goals and vocational skill area(s) in which training is provided. In 
projects which train in vocational skill areas that require basic reading and 
writing (clerical, general office skills, and word processing), assessment of 
literacy* is necessary. In projects which focus on instructional areas that do not 
necessarily require literacy (cosmetology and building trades), only oracy skills** 
need to be measured. 

The th'frd criterion relates to the measureiaent approach taken by a particular 
language proficiency instrument. The ^wo possible approaches are discrete-polar, or 
integrative. If the focus is on the ausessment of individual aspects of language 
such as grammar, vocabulary, e':c., the measurement approach is considered to be 
discrete-point. However, if the orientation is toward assessing an individual's 
functional language ability, the approach is integrative. At the simplest level, 
this latter approach includes the assessment, of an individual's capability to 
listen to a basic conversation and to provide appropriate responses to questions 
regarding name, home address, listing of family members, etc. At a more 
sophisticated level, the assessment could require that an individual participate in 
a simulated Job interview or dialogue appropriate to a work setting. 

Since a general goal at the local level is to assist BVT participants to 
communicate in English in a Job-related setting, a language proficiency instrument 
that takes an integrative approach is recommended. However, it should be noted 
that a discrete-point type instrument may also be appropriately used at the 
project level once an integrative assessment has been made and specific skills 
areas that need to be assessed for diagnostic purposes have been identified. 



literacy refers to reading and writing skills. 

Oracy refers to skills related to listening and speaking. 



30 



-25-" 



The fourth criterion concerns the technical qualities of the instrument. For 
example, it wllil be important to determine whether the testing materials include 
standardized administration and scoring procedures. IThether it is a published or 
project developed instrument, it will be important to revi<=^w validity and 
reliability data in order to judge its adequacy. 

The above four criteria provide a structure for project staff to evaluate and 
select language proficiency assessment instruments. It is important that any 
instrument be carefully evaluated in order to ensure the selection of one which is 
technically sound, and appropriate for the project and the language proficient 
levels of its participants. 

The Basic English Skills Test, developed by the Center for Applied Linguistics, is 
recommended for use as the primary evaluation Instrument for measuring English 
language proficiency. It is discussed in detail In Appendix B of this Guidebook. 
Other measures of English language proficiency are discussed in Appendix C. Local 
projects may choose whichever measures meet their needs. However, the Basic 
English Skills Test is required to be administered by Federal grantees for the 
purposes of the BVT Statistical Summary Report (Appendix A). 
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EXHIBIT 3 

PROGRAM PARTICIPANTS: EXAMPLES OF EVALUATION 
QUESTIONS » VARIABLES AND DATA SOURCES 



Evaluation Questions 



Variables 



Possible 
Data Sources 



1. How many Individuals enter each 
training cycle? 

2. What are the demographic char- 
acteristics of the participants? 



What Is the level of English 
language proficiency of par- 
ticipants before they enter 
the program? 



What Is the level of native 
language proficiency of 
participants before they 
enter the program? 



Number of individuals 
enrolled 



Program 
Records 



Age, Sex, Country of Origin, Program 
Native Language, Years Intake 



of Formal Schooling, 
Achievement of High 
School Diploma or GED Certi- 
ficate, Years 
Lived in U.S., Family 
Status, Number of Dependents. 

Scores on measures of 
English Language 
Proficiency 



Form 



Scores of measures of 
Native Language 
Proficiency 



English 

Language 

Proficiency 

Test Given 

To Participants 

At Intake 

Native 

Language 

Proficiency 

Test Given 

To Participants 

At Intake 



5. What are the vocational skills 
of participants at program 
entry? 

6. How much previous vocational 
training do participants have 
at program entry? 

7. \niat are the Job interests 
and goals of participants at 
program entry? 

8. What are the employment status 
and wages of participants at 
program entry? 



Vocational Skills 



Years of Vocational 
Training; Areas of 
Vocational Training 

Job Interests; 
Job Goals 



Employment Status; 
Public Assistance and 
Unemployment Benefits 
Received; Hourly Wages 
Prior to Program Entry. 



Program Intake 
Form 



Program Intake 
Form 



Program Intake 
Form 



Program Intake 
Form 
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EXHIBIT 4 
SAMPLE OF BVT PROGRAM INTAKE FORM 

Date of Interview: 

Name! Social Security Number: 

Address : 

Telephone Number 

Vocational Training Program: Cycle: 

1. Sex: Male I I Female I I Age: Years 

Date of Birth: 

2. Country of Origin; Native Language: 

3, U.S. Citizen: Yes I I No I I If no, Citizen of 

4, Language(s) Spoken Fluently 

5* LanguageCs) person can read/vrrlte 

6, Number of Years Lived In U.S. Years ' 

7, Years of School Completed: Prior to Arrival In U.S.: Years 

In U.S. : Years 

8, U.St High School Diploma: Yes I I No I I 

9, GED Equivalency: Yes I I No I I 

10# Secondary School Diploma Yes I I No | | 
from Another Country 

!!• Single Parent: Yes I I No I I 
12, Number of Dependents: 
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13. Prior to enrollment, was individual: 
Unemployed: 

Receiving Unemployment Insurance Benefits: 
Receiving Public Assistance Benefits: 
What Type of Benefits are Received: 



Yes No 



Employed 35 or more hours per week: | | | | 

Employed less than 35 hours per week: | | | | 

If Employed, where 

Describe Job — - ■ 



Current Gross Wages Per Hour: $ per hour 

lA. Previous Work History 



15. English Language Skills: 

Date BEST Test Administered: • Form 

BEST Score: 

16. Enrolled in School or Other Training Program Yes I | No 

17. Vocational Skills of Individual: 



18. Prior Vocational Training: 



19. Job Interests and Goals: 
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VI • PROGRAM IMPLEMENTATION 



Tills chapter focuses on documenting how a BVT project is implemented and the 
services which are provided* This includes recruitment, intake and assessment 
activities; vocational and English-language instruction; job counseling and other 
support services; job development; and job placement • The data to be collected 
will focus on what has actually taken place, rather than what was originally 
planned* With proper documentation, the project staff can assess the changes 
needed for future program Improvement • 

Ttie project director and the evaluator should work together to decide on the 
specific activities and services of the BVT program which are to be described or 
documented, and the level of detail desired* Once the Information needs have been 
determined, the project director and evaluator should develop a set of evaluation 
questions which focus the data collection requirements. Sample questions, shown In 
Exhibit 5, cover a wide range of BVT program activities • The project director and 
local evaluator should review these and select the ones which are relevant to their 
program* In addition, modifications and additions to these questions should be 
made to fit the needs of the individual project* 

The evaluator will be responsible for developing the plan*^. and instruments for 
collecting the data to answer the evaluation questions* These data are expected to 
be collected mostly from project records and directly from the project director and 
his/her staff* Two evaluation strategies are available: 

• the evaluator can review project records and interview the project director 
and staff to determine what activities are being implemented and how 
appropriate they are to the needs of the participants; or 

• the project director and staff members can complete written questionnaires 
prepared by the evaluator which address the same issues* 

The latter data collection strategy is probably more cost effective since it 
minimizes the time of the evaluator* 
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EXHIBIT 5 

PROGRAM IMPLEMENTATION: EXAMPLES OF EVALUATION 
QUESTIONS, VARIABLES I AND DATA SOURCES 

PosHlble 

Evaluation Questions Variables Data Sources 



1. What are the objectives of the Program Objectives 
BVT project? Are these objectives 
clearly stated and measurable? 



2. In what vocational areas does 
training take place? 



Vocational Area 



Project 
Application; 
Managemeat Plan 

Project 
Application 
and Records 



3. What language groups are being Languages 
served? 



Intake Form 



4. How long Is each training 
cycle? 

5. How many training cycles are 
taught each year? 

6. l^at Is the total number of hours 
of vocational training provided 
to each participant? 

7. What is the total number of 
hours of English language 
instruction provided to each 
participant? 

8. What are the instructor /partici- 
pant ratios for vocational and 
English language training? 

9. To what extent is enrollment 
meeting the project goals? 



Length of Training 



Number of Training 
Cycles Offered 

Number of Training 
Hours 



Number of Instructional 
Hours ; 

Absentee Rates 



Number of Trainees; 
Number of Instructors 



Project Objectives; 
Number of Participants 



Project 
Schedule 

Project 
Schedule 

Project 
Schedule 



Project 
Schedule ; 
Project Records 



Project Records 



Project 
Application; 
Project 
Records 



10. How is recruitment conducted? 
What criteria are used for 
accepting individuals for 
training? Are criteria appro- 
priate? 

11. How is the project staffed? 
Is staffing appropriate and 
sufficient for project operation.? 



Recruitment Procedures ; 
Enrollment Criteria 



Number of Staff by 
Position 



Project 

Application; 

Observation 



Project 
Records 
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Evaluation Questions 

12. \JhQit are the qualifications of 
the staff? Do staff have 
necessary qualifications to meet 
the needs of the participants In 
both the vocational and English 
language training components? 

13. \Jhat kind of staff development 
and training are provided to 
staff members? Are development 
and training appropriate and 
sufficient? 



Variables 

Background and Experience 
of Staff 



Staff Development 

and Training Activities 



Possible 
Data Sources 

Employment 
Appllcat! ons; 
Staff Resumes 
and Interviews 



Project 

Records; Staff 
Interviews J 
Observations 



14. !7hat specific vocational com- 
petencies are addressed by the 
vocational training program? 



15. What specific English language 
competencies are addressed by the 
training program? Are 
competencies appropriate and 
sufficient? 



Vocational Competencies 



English Language 
Competencies 



Project 
Application; 
Project 
Records 

Project 
Application; 
Project 
Records 



16. What criteria are specified for 
successfully completing tne 
program? Are criteria appro- 
priate? 

17. Does the training curriculum as 
Implemented follow the project 
plan? Is curriculum appropriate? 



Completion Criteria 



18. IThat training methods and 
materials are used? 
Are the methods appropriate? 



Description of Training 



Description of Training; 
Methods and Materials 



Project 
Application; 
Project Records 



Project 

Application; 

Project 

Records; 

Staff 

Interviews 

Project 

Application; 

Staff 

Interviews, 
Observation 



19. To what extent are the native 
language and English used in 
vocational training? Is the 
extent of use of both languages 
appropriate? 

20, l^hat is the extent of coordina- 
tion of the vocational and 
English language training com- 
ponents? Is the coordination 
appropriate and sufficient? 



Percent Use of Native 
Language and English in 
Classroom 



Extent of Coordination 
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21. 



22, 



23. 



24. 



25. 



26. 



Evalua clon Questions 

What Job counseling services 
are provided to participants? 
Are these services appropriate 
and sufficient? 



What job development activities 
are implemented by the staff? 
Are these activities appropriate 
and sufficient? 



What job placement services are 
provided to participants? Are 
these services appropriate and 
sufficient? 



How are employers and the gen- 
eral community involved in the 
training program? Is this 
involvement appropriate and 
sufficient? 



What facilities and equipment 
are used by the program? 
Are the facilities and equip- 
ment appropriate and sufficient? 



What is coat of training per 
participant? 





Possible 


variaDies 


Data Sources 


Description of Job 


Project 


Counseling Services} 


Application; 


Number of Counseling 


Project 


Sessions; Number and 


Ricords; 


Percent of Participants 


Staff 


Counseled; Number of 


Interiews ; 


Counseling Hours Provided 


Observation 


Description of Job 


Project 


Development Activities; 


Application; 


Number of Employers 


Project 


Contacted 


Records; 




Staff 




Interviews ; 




Observation 


Description or Job 


Project 


rjiacemenc services ; 


Application; 


Number and Percent of 


Project 


Participants Served 


Records ; 




Staff and 




Participant 




Interviews ; 




Observation 


Number of Group and 


Project 


maiviQual Meetings Held; 


Application; 


Number of Phone and 


Project 


Written Communications 


Records ; 




Staff and 




Employer 




Interviews. 


Description of Facilities 


Project 


and Equipment 


Application; 




Staff 




Interviews ; 




Observation 


Total Project Funding; 


Project 


Number of Participants 


Budget ; 




Project 




Records 
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VII. PROGRAM OUTCOMES 

ProRram outcome data are used to determine the extent to which a BVT program is 
meetin? its goals and objectives. Generally, these goals and objectives are: 

• stable employment; 

• increased job-related English proficiency: 

• increased earning capabilty; and 

• greater career advancement opportunities. 

The extent to which other goals and objectives are achieved should also be oeasured 
as part of the evaluation. 

To measure the success of a local BVT program, outcome data should be collected at 
three points in time: 

• Program Completion; 

• 90 Days Following Initial Job Placement; and 

• 180 Days Following Initial Job Placement. 

At program completion, data should be collected on vocational and English language 
training outcomes, and on job placements. Trailing outcomes refer to data 
concerning the extent to which participants have learned the vocational knowledge 
and skills taught by the program, and have increased their English language 
proficiency compared to proficiency at program entry. Job placement is concerned 
with placement iii jobs relevant to training, and with wages and benefits received. 

Follow-up data should focus on job retention and advancement, increases in wages, 
satisfaction of former participants with their jobs and with the training which was 
provided by the BVT program, and satisfaction of employers with the former 
participants they employ, especially their vocational and English language skills. 

As with the other evaluation components, the project director and evaluator must 
work jointly to define the scope of the outcome data to be collected. This should 
be accomplished by developing a set of evaluation questions to assess the extent to 
which the project's goals and objectives are met. A list of evaluation questions 
directed at the outcomes of BVT programs Is shown in Exhibit 6. Also shown are the 
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relevant variables which relate to the questions and possible sources from whom 
data may be collected. Project directors and local evaluators should carefully 
examine these questions and select those which are applicable to their program. 
Additional evaluation questions may also be specified which address any special 
issues and concerns of the local program. 

At a minimum, the data required at program completion should be: 

• number of completers; 

• achievement of vocational training objectives and competencies; 

• English language proficiency scores; 

• number of completers placed in jobs related to vocational area in which 
trained; and 

• for those placed, wages per hour. 

Additional data will also be required depending on the final set of evaluation 
questions specified by the project director and evaluator. 

The data on achievement of vocational training objectives and competencies should 
be available from project records. A test or tests of English language proficiency 
should be administered at program completion or exit. The measure (s) shou].d be the 
same as were used at program entry. In this manner, pre/post-test differences can 
be calculated to measure any change In English language proficiency. Data on job 
placements and wages should be obtained from records kept by the job placement 
counselor or collected directly from the former trainees. 

The minimum data required at each of the two follow-up periods are: 

• number employed; 

• wages per hour; 

• number receiving wage increase or higher paying job since initial placement 
following training; and 

• number obtaining job promotions since initial placement following training. 
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As with the program completion data, additional data will be required depending on 
the evaluation questions which are apecified. 

Follow-up data can be collecte».d from the former program participants or from their 
employers. First priority should be given to collecting data from former 
participants, rather than from busy employers. It is suggested that contacts with 
employers to obtain project evaluation data be kept to a minimum « 

The data collecjlon instruments which are generally needed include a program 
completion data sheet for collecting final training and placement data, and 
interview forms or questionnaires for collecting follow-up data from former 
participants and/or from employers. Samples of these data collection instruments 
may be found in Exhibits 7, 8, and 9. These examples may be adapted so that they 
collect the specific data required by each local project. 

Appropriate data collection procedures and record-keeping systems need to be 
developed to ensure that program completion, placement, and follow-up data are 
collected, recorded, and filed. This will ensure that the data are readily 
available for tabulation and summary to meet the time schedule for the evaluation. 
Early planning by the evaluator and project director Is one of the key elements to 
success in any data collection effort. This Is especially true for the development 
of a record -keeping system for the collection of follow-up data. First, a tickler 
file or calendar must be established so that the project staff can keep track of 
when each participant has been placed in a job, and when the 90-day and 180-day 
follow-up periods occur. Without such a system, the staff will not be able to keep 
track of the appropriate time to follow-up each program completer. Second, an 
address file and a placement file for program completers need to be continually 
up-dated. Addressed and stamped post-cards may be given to participants when they 
complete training so they can notify the program of address and job changes in a 
convenient manner. Alternatively, participants can be asked to notify the project 
staff by telephone when they change addresses or jobs« Without these procedures, 
follow-up will be extremely difficult. 
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In general, follow-up data should be collected via telephone interviews with former 
participants. Before completing training, each participant should be told to 
expect these calls, and to keep the project infoifmed of address and telephone 
number changes. On the other hand, collecting follow-up data from employers is a 
very difficult task, as employers are reluctant to give out personnel information, 
and generally are busy and do not like to be bothered. However, if collection of 
data from employers is required, the job development staff should lay the 
groundwork for follow-up by building a good rapport with employers when 
participants are placed. By establishing good relationships at these times, 
follow-up data can be more easily collected. 

The eva3uator, with input from the project director, should be responsible for 
developing the instruments, procedures, and record -keeping systems. The project 
staff, under the supervision of the project director, should have the 
responsibility of collecting and recording the required data. Training the staff 
in all data collection and record-keeping procedures should be the responsibility 
of the evaluator. 
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EXHIBIT 6 



PROGRAM OUTCOMES: EXAMPLES OF EVALUATION 
QUESTIONS, VARIABLES, AND DATA SOURCES 



Evaluation Questions 



Variables 



Possible 
Data Sources 



A* Program Completion and 
Placement 



1* How many participants completed 
the training program? 

2* To what extent did participants 
Increase their English language 
skills? 



3» To what extent did participants 
increase their vocational skills 
and knowledge? 



4* How many completers were 

placed in jobs following train- 
ing? How many were placed in 
training-related jobs, full-time 
and part-time, with and without 
medical benefits? 



Number of Completions 



Pre/Post Measures of 
English Language Skills; 
Achievement of Training 
Objectives 

Achievement of Vocational 
Training Objectives and 
Competencies; Pre/Post 
Measures of Vocational 
Skills and Knowledge 

Number of Completers 
Placed in Jobs 



Program Records 



Program Records 



Completers; 
Program Records 



Completers; 
Program Reports 



5o What Is the average gross wage 
per hour of participants placed 
in jobs? 

6« What Is the average number of 
hours worked per v/eek of 
participants placed In jobs? 

?• How many completers were placed 
in or enrolled in further 
education or training programs 
following completion of training? 

8» Wliat is degree of completers' 
satisfaction with training 
program? 



Wages Per Hour 



Number of Hours Worked 



Number of Completers 
Placed or Enrolled for 
Further Education or 
Training • 



Completers 



Completers 



Completers 



Ratings of Satisfaction Completers 



4 
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Evaluation Questions 



Variables 



Possible 
Data Sources 



B* Folic <^-Up at 90 and 180 Days 
Following Initial Placement 

!• How many program completers were 
employed at 90 days and 180 days 
following Initial placement In 
tralnlng-'related and non-tralnlng 
related 9 full-time and part-time 
Jobs, with and without medical 
benefits? 



Number of Completers 
Employed 



Completers; 
Employers 



2. What Is average gross wage per 
hour of completers at 90 and 180 
days following initial placement? 

3« l^at Is the average number of 
hours worked per week of 
completers, at 90 and 180 days 
following Initial placement? 

4« How many completers are enrolled 
In school or other training 
program at 90 and 180 days 
following initial placement? 

5* How many completers are unemployed 
at 90 and 180 days following 
Initial placement? 

6« How many completers were employed 
at any time during the 90 and 
180-day period following initial 
placement? 

7* How many completers obtained 
job promotions during the 90 
and 180-day period following 
initial placement? 

S. How many completers received wage 
increases or obtained a higher 
paying job during the 90 and 
180-day period following initial 
placement? 



Wages Per Hour 



Number of Hours Worked 



Number of Completers 
Enrolled in School or 
Training Program 



Number of Completers 
Unemployed 



Number of Completers 
Employed at Any Time 
During Follow-nip 
Period 

Number of Completers 
Who Obtained Job 
Promotions or Higher 
Level Jobs 

Number of Completers \^o 
Received Wage Increase 



Completers J 
Employers 



Completers ; 
Employers 



Completers 



Completers 



Completers; 
Employers 



Completers ; 
Employers 



Completers; 
Employers 
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Possible 

Evaluation Questions Variables Data Sources 



9. How satisfied are former Ratings of Job Satisfaction Completers 

participants with the jobs at 
90 and 180 days following 
initial placement? 

10 • How satisfied are employers with Ratings of Employer Simployers 
the vocational and English skills Satisfaction with 
of the program completers they Vocational Skills and 

hired? English Language Skills 
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EXHIBIT 7 

BVT PROGRAM JOB PUCEMENT INPORMAIION 



Name: 



Vocational Training Program:^ 
Program Completion Date J 



A* Placed in Job at Program Completion or Sxit? Yes I I No 



If no, is individual seeking employment? Yes I | No 



If yes: If no, why not? 
!• Name of Company 



Address 

Telephone 



Supervisor's Name_ 
2. Nature of Job: 



ERIC 



3. Is Job Related to Training? I I Yes I I No 

4. Date Hired: 

5. Date Job Begins? 



6. Number of Hours to be Worked Per Week: 

7. Starting Gross Wage Per Hour: 



8. Does Job Include Medical Benefits? 



I I Yes I I No 

B. Enrolled for Further Education at Training Within 30 Days of Program Completion 
or Exit? 



Yes I I No 



Follow-up Dates 



90 Days? 
180 DaysT 
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EXHIBIT 8 



FOLLOW-UP SURVEY OF FORMER BVT PROGRAM PARTICIPANTS 



J I 90-Day Follow-up 

I I 180 -Day Follow-up 



Name: 



Address: 



Telephoae Nuaber: 



Date Individual Completed or Exited Program:^ 
Date this Surv»iy Completed: 



1. Are you currently employed? 



Yes 



No 



(Continue Survey) If no, Have you been employed at any time 

since completion of or exit from 

BVT program? | | Yes 

1 r No 



2. VJho do you work for? 



3. Is this same or different employer from your first job after completing the BVT 

training program? 

Same I I 



Different I I 

If different, name, address and telephone number of employer 



4. Have you been employed continuously since completion of or exit from the 
vocational training program? ______ 

I I Yes 



I I No 

5. Please describe your current job. 
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6* Is your current job: 

Same aa when originally hired 

A promotion from original job 

A demotion from original job | | 

Different job, but neither a promotion nor a demotion | | 

7 4 \7hat is your current gross hourly wage? i per hour 

8. Are you being paid more, less, or the same as the first job you obtained after 
you completed or exited from the vocational training program? 

More I I 

Less I I 

The Same I | 

9* How many hours do you work per week? 



10. Do you receive medical benefits from your employer? 
I I Yes 



I I No 

11* Are there opportunities for you to advance at the place you work? 



Many Opportunities 

Few Opportunities 

No Opportunities 
Explain : 



12. How satisfied are you with the vocational training you received from the BVT 
program? 



Very Satisfied I I 

Satisfied I ' I 

Dissatisfied I I 

Very Dissatisfied I I 
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13. How satisfied are you with the English language Instruct:. on you received from 
the BVT program? 

Very Satisfied I I 

Satisfied I I 

Dissatisfied I I 



Very Dissatisfied I I 

4.4. Would you recommend the BVT program to others? 



Explain: 



Yes 
No 



15. Are you currently enrolled in any school, educational, or vocational training 
program? 

I I Yes 
I I No 



Specify: 



16. How satisfied are you with your present job? 

Like it very much I I 

Like it somewhat | | 

Neither like it nor dislike it I | 

Dislike it somewhat | 

Dislike it a lot |"" "| 

Why? 



17. What changes in the vocational training program do you suggest for the future? 

a) Vocational Training 

b) Engiish Language Training " 

c) other ==— ^ 
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E..HIBIT 9 
EMPLOYER FOLLOW-UP QUESTIONNAIRE 

I I 90-Day Follow-up 
I I 180-Day Follow-up 

Name of Employee: 

Company Name: Type of Business : 



1. Is the above individual currently employed by you or your company? 

I I Yes (Continue with Survey.] 



I I No [If no, ask why not and stop.] 



2. Briefly describe this employee's current job. 



3« How many hours per week does individual work: 

4. Is his/her current job: 

Same as when originally hired 1 

Promotion from original job wicu company \' 

Demotion from original job with company 1 

Different job, but neither a promotion nor demotion 1" 



5. \?hat is employee's present gross hourly wage? i ^per hour. 

6. Is this employee's wage higher, lower, or the same as when originally hired? 

Higher I ' ! 

Lower I I 

The Same I I 

7. Does employee receive medical benefits? 

I I Yes I I No 
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8* Are there opportunities for this employee to advance at year company? 

Many Opportunities I I 

Few Opportunities I I 

No Opportunities I 

Explain : 



9* Compared to other individuals at the same job level, how would you rate this 
employee's job skills? 

Above Average I I 

Average I I 

Below Average I I 

10 • Is this employee's English language skills sufficient to perform his/her 
current job? 

English language skills are sufficient | 
English language skills are a problem | | 

lit Does this employee have the English skills necessary to advance at your company? 

I I Yes 
I I No 

Explain : 



12* In general, how satisfied are you with this employee's work? 

Very Satisfied | I 

Satisfied I I 

Somewhat Satisfied I I 

Dissatisfied I I 

13* What changes, if any, should the BVT program make in its training program to 
enable its trainees to be more valuable employees to your company? Do 
individuals need more training? If so, in what areas? 
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VIII. PRESENTATION OF FINDINGS 



Following data collection, the next steps in the evaluation process involve data 
analysis and preparation of a report. These steps require the expertise of an 
experienced evaluator vho is objective and independent of the project and grantee. 
This is important for the acceptability of the report's findings, conclusions, and 
recommendations . 



The evaluator will be responsible for developing and carrying out a data analysis 
plan which is compatible with the evaluation's goals and audience. To a large 
extent, data will be descriptive in nature and may be presented in narrative and 
tabular format. However, comparisons of pre- and post -measures related to 
English-language proficiency, vocational skills and knowledge, job wages, etc., may 
require more sophisticated techniques. These depend on the nature of the data. 



The data will be analysed answer the evaluation questions specified in the 
evaluation plan. Thus, the analysts will allow the evaluator to: 



- describe the program environment; 

- describe the program participants; 

- describe the program activities and services; 

- describe the outcomes; 

- examine and assess the extent to which the program plan was followed; 

- examine and assess the extent to which the outcomes met the program goals 
and objectives; and 

- examine how the program environment, participants, activities, and services 
affected the extent to which the outcomes were achieved, and how the program 
can be improved to achieve increased success. 

An evaluation report will then: 



• describe the accomplishments of the program, identifying those elements of 
the program that were che most effective; 

• describe elements of the program that were Ineffective and problematic as 
well as areas that need modifications in the future; 

• describe the outcomes or the impact of program services on the participants; 
and 



52 



ERIC 



DIVELOPMSNT ASSOCIATES, INO. 



-47- 



• document how the program environment, activities, and services contributed 
to the accomplishment or lack of accomplishment of the program goals and 
objectives. 

The level of information and documentation suggested in this Guidebook will make 
the report quite useful for making decisions about improving program services, 
instructional strategies, etc., for the future program cycles. In other words, the 
evaluation report is a decision-making and planning too^ for the project director 
and his/her staff. The information presented in the report, as guided by the 
evaluation questions, will support the broad management functions of 
decision -making, program improvement, accountability, and quality control. 

It is important to keep the report's audience in mind. In most cases, this will be 
the project director and agency administrators. These Individuals may not be 
experienced in the technical aspects of data analysis. Therefore, the report 
should present data in a straightforward manner, using tabular presentations to 
help the reader. Interpretations and conclusions drawn from the data and 
Implications for future directions should be discussed in a clear narrative. An 
example of a report outline is provided in Exhibit 10 on the next page. 

In addition to the written report, the evaluator should provide an oral briefing to 
the project director, project staff, and appropriate administrators. This should 
take place after those Individuals have had a chance to read the report. The 
briefing should first Include a short presentation on the most important findings 
and on the conclusions and recommendations drawn by the evaluator. A question and 
answer period following that presentation will allow for discussion of those 
conclusions and recommendations that are most relevant to improving the program In 
order to yield greater benefits to participants, employers, and the 
community-at-large. The project director should then prepare and execute an action 
plan for implementing the appropriate recommendations. In sum, evaluation results 
should always be used for program improvement. Learning from the past to improve 
future programming is always possible and should be a primary management goal. 
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EXHIBIT 10 
ORGANIZATION OF EVALUATION REPORT 



I. EXECUTIVE SUMMARY (3-4 pages) 

- Purpose 

- Procedures 

- Major Findings 

- Conclusions and Recommendations 

II. INTRODUCTION 

- Background of Program 

- Objectives of the Evaluation 

- Summary of Procedures 

III. DESCRIPTION OF PROGRAM ENVIRONMENT 

IV. r»lOGRAM COMPONENTS 

- Descriptions of Program Activities and Services 

- Deviations from Plan in Grant Application 

V. NUMBER AND CHARACTERISTICS OF PARTICIPANTS 

VI. PROGRAM OUTCv ' £S 

- Program Completion and Initial Placement 

- Follow-Up 

VIII. CONCLUSIONS AND RECOMMENDATIONS FOR PROGRAM IMPROVEMENT 
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APPENDIX As BVT Statistical Summary Report 

APPENDIX B: The Basic English Skills Test (B.E.S.T.): 

Measurement of English Language Proficiency 
for the BVT Statistical Summary Report 

APPENDIX C: Revie'*' of Language Proficiency Measures 
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BVT STATISTICAL SUMMARY REPORT 



Background 

The purpose of the BVT Statistical Summary Report is to assist federally-funded BVT 
projects In collecting and reporting data to the federal BVE Program Office* 3y 
standardizing the data to be reported, data can be aggregated across projects. 
This will enable the federal office to assess the overall effectiveness of the 
grant program in meeting Us four major goals; 

• stable employment for limited English proficient adults; 

• increased job-related English language proficiency for limited English 
proficient adults; 

• increased earning capability for limited English proficient adults; and 

• greater career advancement opportunities for limited English proficient 
adults. 

The BVT Statistical Summary Report has been designed to keep the reporting burden 
to a minimum. Only basic program, participant and outcome data are requested. 
This Is important information that, in addition to being useful to the federal 
program office, will be useful to local projects for program planning and 
improvement . 

The Report Is divided into four sections: 

• Section A reports descriptive information on participants, including English 
language proficiency scores on the Basic English Skills Test (BEST). 
Individual scopes on Lhe BEST taken at program intake may be recorded on the 
BEST Data Recording Form (see Appendix B)* Average scores across 
participants are then computed and recorded in Item 11 of Section A. 

• Section B reports completion and initial placement data. Scores of English 
language proficiency at the completion of the program should be recorded on 
the BEST Oata Recording Form and average scores and average pre /post test 
differences across participants are then reported in Item 10 of Section B. 
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• Section C reports follow-up data on former participants at 90 days after 
Initial Job placement. To the extent possible, this follow-up Information 
should be collected directly from the former participants. Tf this is not 
possible, the data should be collected from employers. 

• Section D reports the same follow-up data as Section C, but at 180 days 
after initial placement. 

Instructions 



A 6-month reporting schedule has been established, with reports due within 30 
days of the end of each period. The reporting periods and report deadlines are 
as follows: 



The BVT Statistical Summary Report requests data on each project cycle, separated 
by vocational area, every six months. Thus, every six months a separate 
Statistical Summary Report is to be completed for each vocational training area 
that a project offers, and for each cycle of training. 

The Statistical Summary Report should also be cumulative. Each report should 
provide cumulative totals as of the end of each reporting period . Because local 
BVT projects differ, with each implementing training cycles of varying length, 
projects may not be able to complete all sections of the Statistical Summary Report 
at the end of a particular reporting period. However, in that each report is to be 
cumulative, reports can be updated by providing previously unavailable information 
at the end of subsequent reporting periods. 

For example. If a project Is in mid-cycle when a reporting period ends, only 
participant data will be reported on the Statistical Summary Report (Section A). 
Data on completions, placements, and follow-up ■will not yet be available. With 
cumulative reporting, however, reports will be continually updated in subsequent 
reporting periods. Eventually, all sections of the report will be completed. The 
appropriate box at the beginning of each section should be checked to indicate 
whether the data provided in that section reflect interim or fin'^l results. 
Interim data means that additional data for that section of the report will i>e 



Reporting Period 



Report Deadline 



January 1 - June 30 
July 1 - December 31 



July 31 
January 31 
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available in subsequent reporting periods, while final data indicates that the data 
reported in that section of the report is complete. 

The chart on the next page illustrates the reporting requirements for a 
hypothetical BVT project conducting 3 five-month training cycles. The chart 
indicates those sections of the Statistical Summary Report that can be completed 
for the different training cycles in a given reporting period. This chart is 
shown for illustrative purposes onlyJ Individual projects can use this example to 
help plot their reporting requirements, taking Into account their ovm training 
cycles. 

Summary 

^^hen filling out the BVT Statistical Summary Report, please keep the following 
points in mind: 

• The report requests basic program, participant and outcome data that will be 
useful to your project in program planning and improvement; 

• The report will enable the federal BVE Program Office to aggregate data 
across all of the federal BVT projects and to assess the overall 
effectiveness of the federal grant program; 

• The report calls for cycle reporting, by vocational area, every six months; 
and 

• The report should show cumulative totals; after each reporting period, the 
information provided in the previous report will be updated, until all the 
requested information has been reported. 



ERIC 



59 



DEVELOPMENT ASSOCUTES, INO. 



HLQSTRATION of reporting requirements for niREB CYCLES OF 
A FIVE-HONTH TRAINING COURSE 



Training 
Cycles 



Aug 1 
I 



<— Training Cycle #1- 



Jan 1 
— h- 



June 1 
— H 



Training Cycle 02 ^ < Training Cycle #3— 



Nov 1 

^ — 



Reporting 
Periods 

I 



July 1 — December 31—^ 



Reports 
Due 



Report 
Contents 



January 1 — June 30--^ ^ 



V 

January 31 



Cycle H Report 

Section A: Complete Data 
Section B; Interim Data 



-I 



July 1 — December 31— 
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V 

July 31 



Cycle ll Report 

Section B& C: Complete Data 
Section D: Interim Data 

Cycle l2 Report 

Section A &B: Complete Data 



Cycle <'3 Report 

Section Ai Complete Data 



o 
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January 31 



Cycle tfl Report 
Section Vi Complete Data 

Cycle tf2 Report 

Section C: Complete Data 
Section D; Interim Data 

Cycle tf3 Report 

Section B: Complete Data 
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BVT STATISTICAL SUMMARY REPORT 



BVT Program: 



Vocational Training Area: 



_^Cycle: 



Cycle Start Date: 

Cycle Completion Date: 

Scheduled Number of Weeks of Training in Cycle: 

Average Number of Hours Per Week of Vocational Training: 
Total Numbers of Hour3 of Vocational Training in Cycle: 

Average Number of Hours Per Week of VESL: 
Total Number of Hours of VESL in Cycle: 

Reporting Period: ^to 



Signature of Project Director 



Date 
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A. PROGRAM PARTICIPANTS: 



□Xnterim Data j I 



Pitial Data 



1. Total Number of Participants* 



2. Numbsr of Participants by Age and Sax 

Males Females 

Under 18 ~~ — " 

18 - 21 ~ZZ 

22 - 24 ~~ 

25 - 30 — 

31 - 40 ■ 

41 - 50 
51 and over 



3. Number of Participants by Country of Origin** 

Country Number 



4* Number of Participants by Native Language*-** 

Spanish _____ 

Russian 

Khmer 

Vietnameae 

Farsi 

Lao 



* Participants are those indlvi'fuals who began training In this cycle and 
remained in training for at least the initial two weeks, 

** Country of origin is defined as the country with which an individuaJ 

identifies, either m a birthplace or as a place in which he/she has apcnt some 
significant part of his/her life* 

*** Native language is the language first learned and used at heme. 
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5« Number of Participants by Years of School Completed 



Leilas than, 4 years 
4-8 
9-10 
U - 12 
Over 12 years 



6. dumber of Participants with U.S. High School 
Diploma or 6ED Equivalency 

7* Number of Participants with 
Secondary School Diploma 
from Another Country* 



8* Number of Participants by Years Lived In U.S. 

Less than 1 year 

1 2 years 

3-5 years 

Over 5 years 

9. Number of Participants with 

Zero dependents 

1 dependent ' ____ 

2 dependents 

3-4 dependents 

5 or more dependents 



XO* Number of Participants who are Single Parents 



11. English Language Skills at Program Entry: 

Basic English Skills Test (BEST), Core Section 

Number of Participants with Scaled Scores of 

0 

9 
16 
29 
42 
51 
58 



Average 



Prior to Arrival 

in U.S. In U.S. 



- 8 

- 15 

- 28 

- 41 

- 50 

- 57 

- 64 
65+ 

Score: 



If native country or country of origin does not award a secondary school diploma, 
any diploma or certificate may be counted which is awarded for roughly the same 
number of years of education needed for a high school diploma in U.S. 
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I2« Immediately Prior to Entering Training,'* 



a) Number of participants who were employed 

35 or more hours per week 

b) Number of participants who were employed 

less than 35 hours per week 

c) Number of participants who were enrolled 

in school or other training program 

d) Number of participants who were unemployed 

e) Number of participants who were receiving 

public assistance benefits 

f) Number of participants who were receivin^j 

unemployment insurance benefits 



13. Of those participants who were employed immediately prior to beginning 
training* average gross wage per hour** 



Note that an individual may fall in more than one category. 

Include individuals who worked both full-time and part-time (Item 12a and 12b). 



B. PROGRAM COMPLETIONS AND INITIAL PUCEMENTS* 



I I Interim Data \ 1 Pinal Data 



1. Nufflber of Completers** 

2. Number of Participants Who Left the Program as a 

Non'Completer 

3. Number of Participants Who Remain In Training 

4. Number of Completers Placed In Initial Job 

a. Within 30 Days of Completion 

b. Between 31-45 Days of Completion 

c. Between 46-60 Days of Completion 

d. Between 61-90 Days of Completion 

e. 91 Days or More Following Completion 

Total 



5. Of total In Item B4: 





35 or More Hrs/Wk 


Less Than 35 Hrs/Wk 


Total 


With Medical 
Benefits 


Total 


With Medical 
Benefits 


a. Number Placed In Jobs for 
Which Trained 










b. Number Placed In Other Jobs 











6. Of Completers Placed In Jobs (Item B4 Total), Average 

Starting Gross Wage Per Hour 

7. Of Completers Placed In Jobs (Item B4 Total), Average 

Number of Hours Worked Per Week 



* Report cumulative totals for this cylde at the end of this reporting period. 

A completer Is an individual who meets at least one of the following criteria: 
, - successfully completed the training program; 

- placed in or -obtained a Job with higher wages and /or better advancement 
potential than Job held prior to training; 

- enrolled for further education or advanced training. 
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8. Number of Completers Not Placed In a Job) 
but Seeking Employment 

9« Number of Completers Enrolled for Further 
Education or Training 

10* English Language Skills at Program Completion or Exit: 
Basic English Skills Test (BEST), Core Section 

Number of Completers with Scaled Scores of 



0 




8 


9 




15 


16 




28 


29 




41 


42 




50 


51 




57 


58 




64 






65+ 



Average Participant Score: 

Average Pre /Post Difference: 

Average Number of Days 
Between Pre- and 
Post-Tests 
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C. 90-PAY FOLLOW-UP* 



I \ Interim Data I j Final Data 

1. Potential Number of 90-Day Follow-Ups During 
this Reporting Period 



2. Number o£ Individuals on Which 90-Da7 Follow-Up 
Data were Obtained 



3. Total Number Employed at 90 Days 
Following Initial Placement: 





35 or More Hrs/Wk 


Less Than 35 Hrs/Wk 


Total 


With Medical 
Benefits 


Total 


With Medical 
Benefits 


a. In Vocational Arer for 
Which Trained 










b. In other Type of Job i 











4. For Those Employed at 90 Days Following Program 
Completion* Average Gross Wage Per Hour 



5. For Those Employed at 90 Days Following Program 

Completion* Average Number of Hours Worked Per Week 



6. Number Who Were Employed Continuously During 
the Follow-up Period** 



7. Number Unemployed kt 90 Days Following 
Initial Placement 



8. Of Those In Item #2, 

a. Number Who Obtained Job Promotion or 

Advancement at Any Time During 
the Foll«*T-tiy Period Period 

b. Number Who Received Wage Increase or 

Obtained Higher Paying Job at 

Any Time During the Follow-up Period 



* Employment data are to be collected on Individuals at 90 days following Intlal 
placement. Data on Individuals whose 90th day following Initial placement 
falls before the end of the reporting period are to be Included In this section 
of the report. These Individuals make up the potential number of 90-Day 
Follow-Upc called for In Item CI. Report cumulative totals for this cycle at 
the end of this rei^ortlng period. 

** Employed continuously means employed at all times during the entire 90-day 
period following Initial placement. 



ERIC 



68 



D. 180-DAY FOLLOW-UP* 

I { laterlm Data I | Final Data 



1. Potential Number of 180-Day Follow-tJps During 
thla Reporting Period 



2. Number of Individuals on Which ISO-Day Follow-Up 
Data were Obtained 



4. Total Number Employed at 180 Days 
Following Initial Placement: 





35 or More Hrs/Wk 


Less Than 35 Hrs/Wk 


Total 


With Medical 
Benefits 


Total 


With Medical 
Benefits 


a. In Vocational Area for 
Which Trained 










b. In other Type of Job 











4. For Those Employed at 180 Days Following Initial 

Placement* Average Gross Wage Per Hour 

5. For Those Employed 180 Days Following Initial 

Placement, Average Number of Hours Worked Per Week 



6. Number Who Were Employed Coatinuously During 
the Follow-up Period** 



7. Number Unemployed at 180 Days Following 
Initial Placement 



8. Of Those in Item #2, 

a. Number Who Obtained Job Promotion or 

Advancement at Any Time During 
the Follow-up Period 

b. Number Who Received Wage Increase or 

Obtained Higher Paying Job at 

Any Time During the Follow-up Period 



Employment data are to be collected on individuals at 180 days following 
initial placement. Data on individuals whose 180th day following Initial 
placement falls before the end of the reporting period are to be Included in 
this section of the report. These individuals make up the potential number of 
180-Day Follow-Ups called for In Item Dl. Report cumulative total^^ for this 
cycle at the end of this reporting period. 

Employed continuously means employed at all times during the entire 180-day 
period following Initial placement* 
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MEASUREMENT OP ENGLISH LANGUAGE PROFICIENCY 
FOR THE BVT STATISTICAL SUMMARY REPORT 



70 



DXTELOPMXNT A8i!«O0IATES, INO. 



THE BASIC ENGLISH SKILLS TiSST (B.E.S^T): 
MEASUREMENT OP ENGLISH LANGUAGE PROFICIENCY 
FOR THE BVT STATISTICAL SUMMARY REPORT 



The BVT Statistical Summary Report requires that English language proficiency be 
measured in such a way that data are comparable over all projects. In order to 
accomplish this goal, it has been necessary to identify one measure of English 
language proficiency that can effectively and validly be used across BVT projects 
as a pre/post assessment of growth* However, since no currently available measure 
Is designed to directly assess job-related English language proficiency across 
vocational areas, Instruments that measure an individual's general ability to 
understand and communicate have been judged appropriate to measure the objective. 
The language proficiency measure that has been selected is the Basic English Skills 
Test (B.E.S.T.), which is to be ac^.ministered by all federally-funded projects to 
all project participants a pre- and post-test basis. The summary test scores 
are to be reported on the BVT Statistical Summary Report. 

The Basic English Skills Test 

The E.E.S.T. was developed by the Center for Applied Linguistics (CAL) in 
cooperation with ESL teachers and refugee program administrators. Principal 
funding was provided by the Office of Refugee Resettlement, U.S. Department of 
Health and Human Services. 

The instrument was designed to assess the English language proficiency of limited 
English proficient adults. It Is a competency-based, integrative measure of basic 
functional language skills. Including listening comprehension, fluency, 
communication, reading, and writing. 

The test, which has three equated forms (3, C, and D), has two components. The 
first, the Core Section, is designed to assess basic English oral language 
proficiency; the second, the Literacy Skills Section, is intended to assess reading 
and writing skills. The Core Section is designed as an Individually administered 
10 to 15 minute Intervlev/. In contrast, the Literacy Skills Section is a group or 
individually adtalnlstered test that simply requires a monitor and takes 
approximately one hour to administer. 

k I I " DETXLOPMBNT ASSOCIATES, INO. 
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The two sections of the 3.K.S.T. were field tested with Individuals whose native 
languages included Spanish, Chinese, Vietnamese, Hmong, Lao, Cambodian/Khmer, 
Polish, and others. Thus, It Is an instrument that can validly and effectively be 
used with adults from a wide range of cultural and linguistic backgrounds. 

The topic and linguistic components of the B#E#S*T* were designed to assess 
"survival level'* English competency. They were developed by experts in the area of 
English language proficiency, field tested, and modified based on the findings of 
the field test. An overview of the topic and linguistic components for both the 
Core and Literacy Skills Sections of the B.E.S.T. is presented on the following 
page. 

The B.E.S.T. has three major uses. These are: 

• as a placement iti^^trument for adults entering language training 
courses; 

• as a diagnostic assessment instrument; and 

• as a measure of progress. 

^Thlle these are all valid uses of the B.E.S.T. , the main interest in the instrument 
for purposes of the BVT Statistical Summary Repoit is as a measure of oral English 
language proficiency, a skill necessary for all vocational areas taught by the 
current group of federal BVT projects. On the other hand, this is not true for 
literacy. Thus, only the Core Section of the B.E.S.T. is required to be 
administered and results reported on the Statistical Summary Report. For this 
reason, the Core Section is the focus of discussion here. 



Eakin, E. & Ilyln, T. (in preb^). Basic English Skills Test. In Charles W. 
Stausfield and Karl J Krahuke (Eds.) TESOL Annotated Bibliography of ESL/EFL 
Tests. Washington, D.C. 
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The Core Section has 49 Items and Is Individually administer d as an oral 
Interview. Tl-ie interview Itself takes approximately 10-15 minutes per examinee* d 
cut-off point is set, should an individual not be able to ccrrectly respond to ten 
of the initial set of 13 or 14 questions, depending upoa th< form of the test. 

The Core Section of the B.E.S.T. is designed to provide verbal and visual stimuli 
that elicit responses in English. The first jet of prompts consists of basic 
questions about the individual's name, country of birth, aid present place of 
residence,. The indiridual is also asked to jpell his or hur viame. The other 
prompts are keyed to photographs which require an individuil to respond to a 
question based on the content of the stirauljs, to recognize some basic sight 
vocabulary, and to follow a map representing a neighborhoc d » Reading (recognition 
of simple sight words such as "Closed", "Don't Walk", "Ketw Out", etc,) and writing 
(writing name and address) tasks are also included in the Core Section. These 
items are intended to screen individuals for the Literacy Skills Section and are 
not included in the overall score for the Core Section. 

The key grammatical structures assessed in the Core Section include the simple 
present and progressive tenses, yes/no, wh- questions, and negation. In addition, 
emphasis is givo.n to the language functions of impartlnj, information, seeking 
information, and seeking clarification. 

Specific criteria are given for scoring responses In chree areas: listening 
comprehension, conmunication, and fluency. Sub-s:ale scores are totaled and then 
converted ^o scale.d scores which correspond to Student Performance Levels or SPLs. 

Student Froficiency Levels are general descriptions of a student's language ability 
with respect to listening, oral communication, reading and writinij. The SPLs are 
linked to employment readiness skills and to the qjalities of communication that 
would be evidenced by a non-native English spuakej; in conversatior, with a native 
English speaker. The SPLs and corresponding B.E.S.T. scores are shown on the 
following pages. Because the B.E.S.T. is a measure that assesses language 
proficiency at the lower end of the scale, test scores on the B.E.S.T. cannot be 
linked beyond SPL VII. 
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(\ No ablHIy whatsoever. 

^ B«E«S«T« Core Section, Scaled 


Score - 0-8 




T • Functions minimally, 
^ II at all. in English. 

B.E*S*T* Core Section 

Scaled Score M5 


• Can handle cnly 
very routine entry- 
level jobs that do not 
require oral commu- 
nication, and In which 
all tasks can be easily 
demonalraled. 


• A native English 
speaker used to deal- 
ing with limited English 
speakers can rarely 
communicate with a 
person at this level 
except Ihrdugh 
gestures. 


TT • Functions in a 

very limited way in 
situations related to 
Immediate needs. 

B«E«S*T« Core Section 

Scaled Score » 16-28 


• Can handle only 
routine tnlry-level 
jobs that do not re- 
quire oral communica- 
tion, and in which all 
tasks can be easily 
demonstrated. 


• A native English 
speaker us4d to deal- 
ing with limited English 
speakers will have 
great difficulty com- 
municating with a 
person at this level. 


Tf f * f^unctions with some 
lit difficulty in situations 

related to Immediate 

needs. 

B.B.S.T. Core Section 

Scaled Score « 29-41 


• Can handle routine 
entry-level jobs that 
involve only the most 
basic oral communi- 
cation, and in which 
all tasks can be 
demonstrated. 


• A native English 
speaker us#d ti deal- 
ing with limited Eng- 
lish speakels will have 
great difflclilly 
communicating with a 
person at this level. 

IIIIMM" ■ " 1 ' ■■ 


1 mf * wall salllliy RinSiu 

^ ^ survival needs and a 
few very routine 
social demands. 

B*E«S.T* Core Section 

Scaled Score » 42-50 


* K/ati nanoie eniry- 
level jobs that involve 
some simple oral 

communication, but 
in which tasks can 
also be demonstrated. 


• A native English 
speaker used to deal- 
ing with limited English 
speakers will h5ve 
dillicully cOmmui^i- 
cating with a person 
at this level. 


If • Can satisfy basic sur< 

* vival 1IQDQ9 ttiiu ouiiiii 

limited social 

demands. 

y 

D«E.S.T. Gore Section 

Scaled Score 51-57 


• Can handle jobs 
ana 100 iraining 
that involve following 
simple oral and very 
basiu written Instruc- 
tions but in which 
most tasks can also 
be demonstrated. 


• A native English 
speaKer used 10 aeai- 
ing with limited English 
speakers will have / / 
some dllllculty com- 
municating with a 
person at thh level. 



STUDENT PERFOBMANCE LEVELS 
AND 

CORRESPONDENCE TO B.B.S.T. SCORES 



IKf 1 • Can satisfy most 
V 1 survival nasdt and 

IlllillVU Wwliil 

demands. 

B.E.S.T. Core Section 

Scaled Score «• 58-64 


• Can handle Jobs and 
Job training that 
involve following 
simple oral and writ" 
ten Instructions and 
diagrams. 


• A native English 
speaker not used to 
dealing With limited 
English speakers will 
be able to communi- 
cate with a person at 
this level on familiar 
topics, but with dif- 
ficulty and some 
effort. 


^7TY • CAt\ salisft/ tturulval 

m# IB wmi 0HIIOVT vuiwivm 

VII needs and routine 
work and social 

demands^. 

B«£«S.T* Core Section 

Scaled Score - 654- 


involves foltowing oral 
and simple written 
instructions In familiar 
and some unfamiliar 
situations. 


• A native English 
speaker not used to 
dealing With limited 
English speakers can 
generally communi- 
cate with a person at 
this level on familiar 
topics. 


17|Y| • Can participate ©llac- 
¥ ill lively In social and 
familiar work situa- 
tions. 




• A native English 
speaker not used to 
dealing with limited 
English speakers can 
communicate with a 
person at this level on 
almost all topics. 


TV • Can ^>artlc4pate 

lluentily and accuraieSy 
in practical, social, 

and worK siktfiUons. 




• A native English 
speaker not used to 
dealing with limited 
English speakers can 
communicate uasllv 
with a person at this 
level. 


Y « Abiliiy equal to that 

of a nailvcf speaker 
of the same eocio* 
economic level 

™«. , ^ 
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Technical Qualities of the B.E.S.T . 

Test reliability and validity are well documented in the B.E.S.T. Test Manual 
(1986). High Internal consistency reliability estimates (KR-20) were found for 
both the Core and the Literacy Skills section for Forms B, C, and D ranging from a 
low of .770 (Form D, Total for listening, communication, and fluency) to a high of 
.911 (Form B, Total for listening, communication, and fluency). The reliability 
estimates for reading /writing in the Core Section ranged from .770 (Form D) to ,826 
(Form B). 

High face validity la demonstrated by the fact that the content reflects real-life 
language use tasks. Other evidence of high validity is provided by the 
correlations of test scaled scores and teacher ratings of individuals overall 
language proficiency. 

Scoring reliability was evaluated through an inter-rater scoring procedure. For 
the Core Section, the inter-rater reliability ranged from .992 (Form D, Listening 
Comprehension) to .749 (Form D, Pronunciation). 

Interscale correlations were found to be substantially positive but sufficiently 
different to support the diagnostic use of Individual test subsections. Since the 
B.E.S.T. has been designed to reflect actual language-use needed to function 
Independently in the United States, the ability of an individual to perform these 
functions is Important information that can be used to diagnose English language 
competencies for placement, to provide Information for general planning purposes, 
and to serve as a measure of individual progress. 

Guidelines for Using the B.E.S.T . 

The B.E.S.T. should be administered on a pre- and post-test basis to each BVT 
project participant. The pre-test should be adminlt^-ered prior to the beginning of 
training or as soon airter training begins as possible (within the first two 
weeks). The post-test should be administered at the end of t alning or, at least, 
during the last week of training. Any of the three equated forms (B,C, or D) may 
be used, but the same form should not be used as both a pre- and post -test for any 
individual. Trainees should not be told that a post-test similar to the pre-test 
will be given at the md of training . 

^ ' ^S=ga; ssaaass BETELOPMINT ASSOCIATES, INO. 
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The pre-test may be administered by any project staff person who Is English 
proficient, Including a participant's potential VESL or vocational education 
teacher. These staff members need to be adequately trained in the standardized 
administration and scoring procedures before they begin any testing • For the 
post-test, it is not advisable for a participant's teacher, particularly the 
individual's VESL teacher, to administer the test. In order to give the project 
flexibility in scheduling pre-testing and post -testing, it is advisable that 
several individuals on the staff be trained to administer the B.E.S.T. This will 
also give the project alternate test administrators should a staff member leave, 
and will ensure that there will be staff available to train newly hired individuals. 

Testing should take place in a quiet room where there will b'i minimal 
interruptions. In order to administer the test comfortably, a desk or worktable 
needs to be available where the test administrator and tb/ trainee can face each 
other. A flat surface is needed in order to be able to manipulate test materials. 

The specific materials required to administer the B.E.S.T. include the 
Administrator's Manual (1986), the Core Section Picture Booklet > the Core Section 
Scoring Booklet, the Core Section Scoring Sheets, currency (3 one dollar bills, 2 
quarters, 2 dimes, 2 nickels, and 4 pennies), and sharpened pencils.* 

The test administrator should follow all the standardized Instructions in the test 
administration manual. In general, the test administrator should maintain a 
neutral and natural demeanor; state items as written; repeat Instructions or 
directions only once; stop at the specified cut-off, depending upon the form; and 
provide prompts, as appropriate, on fluency items. 

Scoring of the examinee's responses are recorded in the Core Section Scoring 
Booklet or on the Core Section Scoring Sheet as the test is being administered. 
Responses are assessed in the following areas: listening comprehension. 



Test materials are available from the Center for Applied Linguistics, 
1118 22nd St., N.W., Washington, D.C. 20037. 
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communication, fluency, and reading and writing. Although the scoring booklet 
provides examples of correct responses. It is very important that the test 
administrator be trained in scoring student responses • 

Once the testing session is completed, total scores are computed for each skill 
area. The test administrator then adds the scores for listening comprehension, 
communication, and fluency. Depending upon the form of the test, the total Core 
Section score is then converted co a scaled score which can be related to an SPL 
level* 

The reading /writing component i« not part of the Core Section score. It is bailed 
on the recognition of sight vocabulary items and on the completed bio-data item* 
Pronunciation is rated globally and is based on overall comprehensibility. These 
two scores may be calculated as general measures of language proficiency, but are 
not required for the BVT Statistical Summary Rciport. 

Reporting B>E>S>T> Data on the BVT Statistical Summary Report > 

A summary of the pretest scores for all parti lipants in a training cycle are to be 
recorded on Part A of the BVT Statistical Suiuaiary Report. A summary of post-test 
scores, average pre/post differences, and average number of calendar days between 
teats are to be recorded on Part B of the Report. 

In order to summarize the data for the BVT Statistical Report, it will be necessary 
for project staff to record and maintain individual student scores. For this 
purpose, the B.E.S.T. Data Recording Form has been developed (see attached form). 
This form is provided as a management tool for BVT project staff and can be used 
exactly as shown or revised to be more project specific, \7hether this form or a 
project specific form is used, the important data to be recorded includes the date 
of the test, a record of the B.E.S.T. form (B, C, or D) administered to the 
trainee, and the individual scaled score. This information is to be recorded for 
both the pre- and post-tests and will provide the raw data for the language 
proficiency items on the BVT Statistical Summary Report. 

SO 



B.E.SJ. DATA RECORDING FORH 

Program; 

Vocational Training Area: Cycle: 





PRE-TEST 


DACT TCCT 

POST-TEST 


nnr* inA<*T nt rr*r*ni*fti/^r rata 

PRE/POST DIFrERENCE DATA 


Name of Participant 


ronn uciifC ui olciicu 
Used Test Score 


Fnrin DatiP of Srfllpd 

Used Test Score 


Calendar Days p^e/post 


















• 








• 




































• 
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Suttmary Data 






Average No. Average 
of Days Difference 


Average Scaled Score 

t 1 . 1 1 A 1 ■ 1 'm' 


Average Scaled Score j j 
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REVIEW OF LANGUAGE PROFICIENCY MEASURES 



In addition to the Basic English Skills Test (see Appendix B), various other 
measures of English language proficiency are available for use by BVT projects. 
These measures are reviewed below. They are divided Into two categories: 
commercially published Instruments and program developed Instruments. 



Commercially developed language proficiency Instruments which may be appropriate 
for BVT projects represent measures that can be used ko Independently assess all 
areas of language proficiency Including listening, speaking, reading and writing. 
To provide an overview of these measures, each Is described relative to Its 
purpose, stated objectives, and technical qualities. 

1. Bilingual Vocational Oral Proficiency Test (BVOPT) 

The BVOPT, published by Melton Peninsula (1981), is a criterion-referenced 
test, with the criteria based on "an extensive study of the actual language of 
the vocational setting." It is stated that each item on the the test is 
referenced to a set of communicative criteria and a set of linguistic 
criteria. However, it is also stated that while a ^'common set of communicative 
and linguistic criteria emerged from all the vocational areas examined - there 
was little commonality of vocabulary." Thus, in developing the BVOPT, rather 
than being specific to a range of vocational areas, the test Items were based 
on high frequency words. Moreover, in order to appeal to a wide audience, 
topics used for test items are reflective of issues that would be common to any 
recent immigrant (e.g. food shopping, cleaning house, etc.). 



A. COMMERCIALLY PUBLISHED ENGLISH LANGUAGE PROFICIENCY MEASURES 
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An attempt was also made to link test Items vlth linguistic structures. 
However, no provision is made to identify the Unguis t:lc structures used by 
students t Thus, It Is difficult to assess whether an individual student can 
control any of the Identified structures • 

According to the test author,* the BVOPT has two purposes; 

• To screen applicants for enrollment in programs designed for adult 
vocational training; and 

• Ti assess gain in English proficiency achieved while participating In a 
vocational education program* 

It should be noted that although the test purposes are stated, there is no 
further discussion of how the BVOPT meets these purposes. 

The BVOPT is an integrative test that is designed to measure listening 
(receptive) and speaking (productive) skills. The test is individually 
administered, and has four subtests that require verbal or physical responses. 
The test sections include: Question/Answer, Open-ended Interview, Elicited 
Imitation, and Imperatives. Two alternate equivalent forms intended to be used 
for pre- and post -test purposes are available. 

The Questlon/Aiiswer section consists of a set of oral interview questions which 
Increase in difficulty. This section is scored on a scale of 0 
(inappropriate) to 1 (appropriate). The Open-Ended interview is used to obtain 
a sample of the examinee *s speech. The speech is elicited from the students 
through an exercise which requires them to view and discuss a set of 
photographs. Based on the quality of the speech sample, the test administrator 
rates the individual's English language proficiency as "high," "merilam," or 
"low" where each category is further subdivided into three levels. The 
Elicited Imitation requires that the examinee repeat a statement given by the 
administrator. Although a reference is not cited, this item type is based on 



*Mary Galvan is the major BVOPT author (Messerschmitt , in press). 
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Natallclo'r (1976) research af timing the utility of sentence repetition as a 
language proficiency assessment technique (Messerschmlt, in press). 
Messerschmit indicates that she Informally tested the validity of the elicited 
imitation technique with "native and high level non-native speakers" and found 
"clear differences in success ... tl. is, supporting the validity of this item 
type." Scoring is on a scale where "1" means perfect or intelligible and "0" 
represents a response that is unintelligible or not provided. 

The final subtest, Imperatives, consists of a set of commands which are 
intended to assess aural comprehension. This subtest requires that the 
students physically manipulate objects in response to specific commands. The 
responses are rated "y" indicating that the examinee does exactly what Is 
requested, "n" if the examinee does not do exactly what Is requested, or "0" 
when the ecaminee gives no response. 

Although the manual indicates that two types of scoring are possible, only one 
three-stage procedure is described. Raw scores are calculated for each 
subtest. The raw scores are then converted to adjusted scores. The total 
converted score is obtained by adding the four adjusted subtest scores. 
Individuals who score below 199 are considered "low level speakers"; 
"medium-level speakers" achieve a converted score between 200 to 299; and 
"high-level speakers" achieve a converted score above 300. 

The test administrators' manual (Melton Peninsula, 1981) indicates that the 
test administrators "must be literate in English as well as have native or 
near-native proficiency in English" am' that they "should also be able to give 
explanations and directions In the student's native language to reduce tension 
ard frustration." It should be noted that these instructions could be 
problematic in cases where teachers are not bilingual or where students from a 
variety of language backgrounds are enrolled. 
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The administrator's manual also states that "no special training is needed for 
test administrators" (Melton Peninsula, 1981), Although instructions are 
clear, it would seem advisable to have a more experienced person familiariz;e a 
less experienc i individual with the instrument. In addition, to further 
ensure standardization, where several Individuals may be called on to 
administer the instrument, it may be useful to have a native English speaker 
record test items on a cassette « 

Content validity of the BVOPT was established by having respected authorities 
evaluate the extent to which the test measured what it purported to measure < 
It was also reviewed for cultural bias by other experts • The administrators' 
manual Indicates that construct validity was also establishec.* However, 
supporting documentation is not provided • Internal consistency is also 
discussed* Although tables are presented, clear documentation la not given to 
support the assertion that "subtests correlate with the test as a whole, and 
the two forms correlate with each other at a statistically-respectable level" 
(Melton Peninsula, 1981) • 

An argument is made that inter-rater reliability was not assessed because no 
attempt has been made to establish norms* In the same discussion, the 
statement is made that in the field test there were differences in the way 
raters scored students* However, "the staff concluded that differences between 
raters lay in interpretation of responses rather than in the questions 
themselves" (Melton P^^nlnsula, 1981)* The authors dismiss the problem of 
differences between raters by stating that: 

Great care has been taken in admonishing test administrators to 
evaluate responses on the basis of appropriateness and 
understandability rather than precise linguistic correctness* 
The staff is satisfied that trained and untrained test 
administrators get approximately the same results (Melton 
Peninsula, 1981)* 



*The principal author supports the need for training* She recommends that one to 
one and a half hours be allot ed for training. 
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It would seem from the above statement that Inter-rater reliability issues were 
of concern during the field test. However, the issue was not explored. 

2. Diagnostic Test for Students of ESL * 

The Diagnostic Test for Students of ESL (Davis, 1953) is a grammar-based 
measure designed to assess knowledge of English structUL-ea and idiomatic 
vocabulary. It is a 75-item, multiple choice, group administered test that is 
designed as a placement measure for adults in ESL courses. The test has 
commonly been used to assess applicants from non-English speaking European 
countries for admission to American universities. Tlie test ia available at 
three different levels. Specifically, the measure Is desltjiied to asaeas 
lexical comprehension and written syntactic comprehension t 

Data on test reliability and validity are not available ► The author provides 
suggested cut-off scores to Indicate performance level. A Speannan-'Brown 
split-half reliability coefficient of .96 was reported by Manuel (1959). 

The test may be used both as part of the intake placement process and as a 
component of a program's overall evaluation. For this U car purpose, it is 
administered using dlff^'rent test leveljj mid-cycle and at the completion of the 
project to assess student progress. It should b« noted that the project 
administrators where this ta&t Is used feel that the Instruraent Is u/neful as 
training for clerical students who need to be prepared to take slmilti.r language 
proficiency measures as part of the job application procest*. 

3. English language Skills Assessment 

The ELSA (Ilyln, Doherty, hte., and Levy, 1980) is aa integrative 
criterion-referenced placement teat for jidults- The ELSA Is daslgne-.d to 
measure meaning in context, as well aa grammatical ability" (Doherty h Ilyln, 
1981). The test, published by fls.rper & Row Publishers, consists of a sarles of 



Once available through McGraw-Hill, the Diagnostic Test for Students of ESL is 
no longer published « 
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reading passages In a multiple choice CLCZE format. It was specifically 
developed to assess different levels of ESL proficiency of "adult resident 
immigrants" (Doherty & Uyin, 1981). 

The ELSA is Used on the syllabi of the adult education project at San 
Francisco Community College Centers. Although the primary purpose of the ELSA 
l3 placement > the authors state that it can also be used to measure achievement 
over time and to "provide comparison data on levels of instructioa at Adult 
Centers" (Doherty & Ilyln, 1981). 

'i.'he ELSA, which Is group administfired, is intended to discriminate three levels 
of English language proficiency - beginner, intermediate, and advanced. Six 
forms of the test, each with 25 items, are available. The beginning and 
intermediate levels each have two forms; one is in a conversation format and 
the other in a narrative format. It should be noted that despite the different 
formats there seems to be an assufliptlon that the two tests are comparable; no 
discussion of this Is found in the 11LSA Manual. At the advanced level only one 
form in a narrative format is available. 

Teat scores are converted into "ELSA levels" of proficiency ^ The scalci ranges 
from 100-800, where beginning level students score 300 or less, students at the 
intermediate level score between 300 and 500 acid advanced students score 500 to 
800. THiQ authors note that since ELSA levels are based on the San Francisco 
Community College Center's curriculum, they may he inappropriate for student 
placement in other programs (Doherty & Ilyln, 1981). The authors state that in 
such cases "programs should establish their c^ra levels," 

Reported test reliability coefficients are higti^ Validity was also established 
through eoiplrlcal studies. The ELSA was crlginaily developed as a substitute 
for a lengthy ba<:tf>ry of tests to provide a isi^.asurement of global English 
language proficiency. To v&lidate the KL3A for this purpose, test scores were 
correlated with othsr measures including thft Coasprehensive Language Test for 
Speakers of English as a Sscond Language (CELT) (Karris ft Palmer, 1970), the 
Llstesulng Comprehension Test (LCBT.) (IlyLn, 1981), the Michi£;an Test of 
English Languajje Proficieney (MTELP) (Ufllversity of Michigan 1962-1964), 



Structures Tests-English Language (STEL) (B'.ist & Ilyln, 1976), and thf. Tent of 
English as a Foreign Language (TOKFL) (£TS, version oot specified). High 
correlations ranging from ,7.3 to .64 Tfith all but the listening teats (LCPT 
and CELT listening) ware fouu«l. Since t isae reeultfj did not support uee of :he 
ELSA. as an overall measure oi: lang'jago proficiency, the test authors indicate 
that the tost would best be umi to asoess reading and grammar and that it 
would be neceis&ary to use 3e]>arate cLeasures to aaseas other skill areau. 

I'l^g.U ^gkjya a Second Language Qyal Assessment (ESLOA) 

The ESLOA.; available throuj»',r f.he Literacy Volunteers of America (X978), Is 
designed to ttiewaure Englisl'. oral communicative and aural comprehension skills 
of students from non-English language backgrounds. The test is designed to 
assess four levels of ESL proficiency. Level 1 requires no verbal response; 
students are asked only to point to the correct p^cturs, Ijevel II is designed 
to aasess survival vocabalary; Level II J. to measare e.n individual's? ability to 
answer information quea'^ions, and Level IV to asaeae a person's ability to 
accurately use Enijlish forma. 

Ceiling an-l floor proficiency levels are established for each leveJ.. No otl'ver 
information regarding score interpretation is provided. Other techr.ical dat^a 
were also not available. 

Lai.uuage Assessraeut Scale s 11 (LAS II) 

»w»«r.»»» rn iiM - — •n iiWiii«i«i uin »lnwiM i <ii«^««i» till I V. I m^^t 

The LAS II (Duncan and Da Avila, 1981) "io a convarijent assessment mea^iiire" 
designed to assess the oral English language skills of students in grades 6 
throu{j;h 12+. Tiie aim of the IAS II j, published by the Unguametrics Groap, is 
to provide "an overall picture of language proficiency baaed on a student's 
performance on four linguistic subaysfetRs" (Duncan & Dv» Avlla, 198J.), Thoae 
subcomponents include phoncmica (knowledge of phonemes, stress, lythm and 
intonation), lexicon (coajprehension of words), syntax (rules for compreheudinj^ 
and constructing meaningful utterances), and pragmatics Cthe appropriate use of 
language to obtain specific goals). The IAS II, which ha.D two available toxms, 
is Individually administered. 
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The Individual's knowledge of phonemlcs Is measured In two ways, through a 
minimal pairs subtest and via a test which assesses an Individual's ability to 
repeat specific English sounds. Knowledge of lexical Items selocted from the 
Thorndlke-Lorge word list Is assessed through the oral Identification of 
pictured items. Understanding of English syntax is measured in two ways 
utilizing both an aural comprehension and an oral production measure. Aural 
comprehension is assessed via a discreet point approach where a student listens 
to a recorded sentence on a cassette and then is asked to select the one that 
best illustrates the sentence from among three pictures. The sentence items 
Include a range of specific syntactical markers such as plurals, possessives, 
negatives, tense, number, active versus passive voice, comparative adjectives 
etc. To assess oral production, a story retelling procedure is used to elicit 
language samples. Based on the results, qualitative holistic ratings are made 
according to one of five levels, with level 1 representing minimal proficiency 
and level 5 an "articulate native speaker." 

Raw scores for each subsection are converted into weighted scores, with greater 
weight placed on the story retelling task. The weighted score is then 
converted to a level where levels 1 through 2 represent a non -native speaker of 
the language, level 3 a limited speaker, and levels 4 to 5 a fluent speaker. 

Evaluation of pragmatics is assessed through an optional observation section 
which includes a range of sociolinguistic tasks pertinent to success in tae 
school environment. The observation is intended to be completed by a teacher 
or other adult who knows the student well (not the examiner). It is 
recommended that this optional measure be administered to students whose scores 
fall in the "grey area," i.e., within one standard error (2 points) of the cut 
off point for a particular proficiency level. 7.1 theae cases, the additional 
information is expected to reduce mlsclasolf ication errors. Extensive guidance 
for scoring and score interpretation is provided in the Scoring and 
Interpretation Manual for Language Assessment Scales (Duncan & De Avila, 
1981). Exercises to establish inter-rater reliability are provided as are 
sample protocols at each level. 
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It should be noted that although the authors suggest that the LAS II is 
suitable for students In grades 6 through 12+, the content is most appropriate 
for students at the junior high school level, adequate for assessment of 
students at the senior high school level, and least relevant at the adult level* 

LAS II technical information is well documented • Validity studies for LAS II 
were based on agreement between test results and the judgments of bilingual 
adults used to separate individuals who are monolingual English speaking from 
Spanish spenkers* Although the technical data suggests that the test is 
reasonably reliable, with such broad levels of classification "oiany questions 
about validity for the five levels of the test go unanswered" (Groarty, in 
press)* 

Listening^ Comprehension Picture Test (LCPT) 

The LCPT (Ilyin, 1981) is designed to measure listening comprehension of basic 
English structures of non -native speakers of English without requiring reading 
or writing skills^ This multiple choice test, available through Newbury House 
Publishers, is group administered • It is designed as a placement measure for 
beginning and intermediate adult students of ESL, or to place low beginners in 
college intensive English programs* The authors note that the test is not 
suitable for "students with no educational experience" (Ilyin & Rubin, 1981)* 
The LCPT utilizes Interrogative questions to elicit responses from examinees 
who are asked to select from among five choices* 

Although the LCPT is intended to be used as a placement me^.sure, it is also 
described as a diagnostic instrument and as a measure of achievement to assess 
student progress in English language development • Its use as a diagnostic 
instrument is greatly restricted because of the limited number of grammatical 
structures that provide the basis for creating a student profile of strengths 
and weaknesses* Based on the data presented in the Technical Guide, the LCPT 
should also be used cautiously as a measure of achievement* Abdelal (in 
press) observes that 
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mean gains of a group can only be loosely Interpreted. The 
practice and memory effects can be large especially with beginning 
students, if suitable time has not elapsed between pre- and post 
test. Mean gains may only reflect greater familiarity with the 
test method and content (p«5). 

Validity and reliability information is provided in the LCPT Technical Guide 
(Ilyin & Rubin, 1981). Validity data consist of correlations with other 
measures including the ELSA and STEL which were authored .id co-authored, 
respectively by Ilyin. Pearson product correlations ranged from .48 to •66% 
Correlations with the Michigan Test of English Language Proficiency (MTELP) and 
the English Language Institute (ELI)-Aural were r**.19 and r=.39, respectively. 
The author attributes the low correlations with these measures to the 
difficulty of the test for students enrolled in the 100-500 level course at the 
San Francisco Community College Center* 



Content validity Is justified by the statement that "instructors reported that 
the tests contained material that was taught in levels 100 to 400 and was 
appropriate for those levels" (Ilyin & Rubin, 1981). It should be noted that 
the LCPT was not found to discriminate in placing students into levels between 
500-600. Evidence of validity is claimed by virtue of the differences in 
reported means for levels 100-400 on a developmental form of the test. Abdelal 
(in press) suggests that the evidence provided is, in fact, not very strong. 

The fact that the group means are different with four levels is 
not sufficient evidence that the test discriminates well at these 
levels. An average difference of one or two correctly answered 
items does not seem sufficient to assume that the test is 
discriminating well at these levels (p. 4). 

Test reliability is quite high, ranging from .81 to .98. However, Abdelal 
points out that several admlnlstrativ<i procedures could affect reliability. 
These include flexibility in translat:><? test instructions, timing between test 
questions, and timing between administering the last 8 items to students who 
score above 40%. 



7. Structure Tests - English Language (STEL) 

The STEL (Best and Ilyin, 1976) is designed to measure knowledge of syntactic 
structures and vocabulary in English. It is a group administered, multiple 
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choice test available at three difficulty levels with alternate forms. It la 
described as an Instrument that can be used as a measure of achievement as well 
as a placemenc test. 

Both reliability and validity information are available for the the STEL. 
Reliability coefficients range from .86 to .90. Parallel form reliability is 
also reported. Content validity was established by comparing the content of 
the ESL courses taught at a particular adult school in San Francisco. 
Predictive validity was also established using expert teacher judgment where 
ESL teachers predicted whether students would fall in the top 25, middle 50 or 
bottom 25 percent e Differences between the groupo were in the expected 
direction and were statistically significant. 

Technical information needed to Interpret test results is not readily 

accessible in one document. The Table of Equivalence Scores based on the San 

Francisco Adult Programs are found in the STEL answer key vrhich is part of the 

STEL package. An additional source of information is the publication, "Newbury 

it 

Linguistic Grading Scale," which provides specific informatici about the 
required content to be mastered at each level. 

It should be noted that the STEL reflects a structural linguistic perspective 
which focuses the assessment on only one aspect of language proficiency - 
ability to correctly identify English language structures. In general, the 
strengths of this measure include its ease of administration and its 
inexpensive multiple choice format. 



*Thls publication is no longer in print. 
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8r Test of Adult Basic Education (TABE) 

The TABE, published by CTB/MC Graw-Hlll (1967, 1976, 1986), Is designed to 
measure reading, vocabulary, reading comprehension, language mechanics, 
language expression, spelling, mathematics computation, and mathematics 
concepts and application* It Is Intended to be used for adults with "limited 
education and from various educational backgrounds.** The test Is available at 
three levels E (easy), M (medium), and D (difficult). The E level test is 
Intended for adults with severe educational limitations or for individuals from 
culturally disadvantaged backgrounds. Levels M and D are adapted from the 
elementary and junior high school levels of the California Achievement Test 
(CAT). The TABE, which has tm alternate forms, can be group administered. If 
administered in its entirety, the test takes approximately 3-1/2 hours. 
Practice exercises as well as a locator test are available. The locator test 
is designed to determine the TABE level that should be administered to a 
particular individual. 

Scores for individual test sections are based on the number of correct items. 
The total raw test score is based on the addition of the section scores. The 
total score la then plotted on a student profile sheet that indicates grade 
level achieved. Diagnostic analysis of individual section errors is also 
possible so that the test results can be used for curriculum planning. 

No Information is available about test reliability. Content validity is based 
on the fact that Item selection is based on procedures established for the 
California Achievement Test (CAT). 

The TABE may be used both as a screening measure and as a measure of 

achievement over time. As part of the screening process the results may be 

used to identify whether an individual is literate in English and familiar with 

basic arithmetic concepts. Used in this way, the TABE is considered to be an 

adequate screening instrument. It should be noted, however, that users are not 

confident of the measure as a means oz assessing language growth for adults 

from non-English speaking backgrouncis. It has been found that when 

Individuals are retested they often demonstrate no growth and sometimes even 

score less well than they originally did even though their teachers * informal 

and formal assessments Indicate that there has been substantial progress In 
their job-related English language proficiency. 
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B. PROJECT-DEVELOPED AND PUBLIC DOMAIN ENGLISH 
LANGUAGE PROFICIENCY INSTRUMENTS 



The China Institute In America (CIA) utilizes project developed measures for 
Intake 9 placement, and measurement of overall performance* Their measures Include 
the Chinese Chefs' English Screening Test, the Oral Proficiency Placement Test, and 
a performance-based English test. 

As part of the Intake process. Individuals are assessed on the Chinese Chefs 
English Screening Test. This measure was designed to distinguish the oral 
proficiency of Individuals at the low end of the scale In ways not possible In 
commercially available tests* The test, which is group administered, has six 
sections. They include assessment of the following skills: letter recognition, 
ordering, word recognition, creation of antonyms, placement of phrases or nouns 
within a sentence, completion of correct verb forms within a sentence, ability to 
comprehend complex statements or questions, translation of the statements or 
questions into Chinese, and student responses based on questions of varying 
complexity* 

The Oral Proficiency Placement Test, used as a pre/post measure, consists of the 
following questions: 

1. Do you understand English? 

2. What's your name? 

3. Are you Japanese? 

4. \ifhere were you born? 

5. When did you come to America? 

6. Is Sweet and Sour Pork spicy? 
7* Did you work yesterday? 

8. What will you do tomorrow? 

9. !^hat Chinese food can you make? 

10. What do you need to make Dong Ting Shrimp? 

At the instructors* discretion, higher level questions can be added when it is felt 
that an individual has progressed sufficienCly to respond to more difficult 
questiond than those prescribed. Scoring is based on a four point scale ranging 
from '*no response" to "acceptable." 

A performance-based English test is administered during the 5th week of the 
program. The assessment requires that a student describe a recipe in English. 
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In sum, the Instruments developed by CIA include screening, placement, and 
performance"-based pre/post -evaluation measures. 

The BEST project, at Oakton Community College, also utilizes project-developed 
language proficiency measures for both intake and project evaluation* As part of 
their screening procedures, applicants who make initial contact via telephone are 
rated on a six-point scale ranging from "very poor" to '^excellent. " ^Jhen 
applicants formally apply to the program, they take the Language Proficiency 
Interview (LPI), a modified Foreign Service Institute (FSI) type measure as well as 
a project developed CLOZE test. 

The LPI interview, which is a direct measure of speaking proficiency. Is intended 
to assess comprehension and speaking ability. It is designed to "elicit natural, 
flowing conversation." Standardized FSI scoring procedures were modified so that 
the LPI is scored to place equal emphasis on fluency and grammar. The project 
director emphasized the importance of the test administrator's experience with the 
measures in validly scoring an individual's performance; it Is felt that the 
administrator should be trained and that, it possible, the same person should 
administer the pre- and posttest. 

The CLOZE test consists of two paragraphs from an air conditioning/refrigeration/ 
heating textbook. The passage is presented so that every nth word is deleted for a 
total of 20 missing words. Students are instructed to complete the passage. Each 
response represents a score of 5 for a potential total score of 100. 

The test results are used initially to compare applicant skills for screening 
purposes. Once an individual coiv^pletes the program, the test is readministered as 
part of the overall project evaluation. 

As a general rule, it has been found that Individuals who score 20 or lower will 
have difficulty with the project curriculum. Individuals who score above 70 are 
guided to apply to other vocational education projects because experience has shown 
that such individuals may feel bored in a BVT project that has a strong emphasis on 
English language development. 
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One BVT projact utilizes an observation scale — the Student Oral Language 
Observation Matrix (SOLOM), The SOLOM, which Is based on the Foreign Service 
Institute (FSI) oral proficiency Interview, Is In the public domain* The SOLOM was 
developed within Ine San Jose, California Unified School District to assess the 
oral English proficiency of elementary and secondary students. It is Intended as a 
measure that provides information about an individual's ability to comprehend both 
formal, instructional language, as well as informal, conversational situations • 

In contrast to the Foreign Service Institute (FSI) Oral Proficiency Interview which 
was designed to assess foreign lai.guage proficiency of adults in the foreign 
service, the SOLOM was designed for use within a classroom context. Thus, there 
are several notable differences between the SOLOM and the FSI, First, the SOLOM 
ratings are based on teacher observation of Individual students* use of oral 
English across a variety of situations rather than on simulated situation ? where an 
Individual Is asked to perform Independent of the context. Second, the components 
of language assessed were revised for the SOLOM to include pronunciation rather 
than accent. Finally, the verbal descriptions attached to each level of 
proficiency in a given area of the FSI Oral Interview were adapted for the specific 
purpose of evaluating elementary and secondary limited English proficient youth 
language proficiency rather than that of adults. 

Used in the BVT project, the ratings of proficiency are applied to limited English 
proflclenct adults. They are rated on a five point scale in the areas of oral 
comprehension, fluency, vocabulary, pronunciation, and grammar. The rating level 
characterizations are general providing a description of a student •s global 
proficiency rather than an analysis of an individual's grasp of discrete aspects of 
the language, A lower score Indicates less English language proficiency, A score 
of 5 in any one area indicates native-like proficiency in English, equal to that of 
a native English speaking individual, A total SOLOM score is based on the sum of 
the rating in each category with a maximum possible score of 25, 

Specific validity and reliability data do not accompany the measure. However 
Zehler (1986) reports on two studies which attest to the technical qualities of the 
SOLOM* Clark (1978) and Adams (1978) in studies using the FSI Oral Interview 
provide evidence that there are high levels of agreement among different 
individuals rating the same person, 

loo 
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The SOLOM is similar in format and approach to the LPI instrument used at Oakton 
Community College • Like the SOLOM, it is intended to be used by a teacher who has 
worked with an individual in a classroom context and who has had other interactions 
that involve language use* 



C. SUMMARY 



The preceding discussion presented a number of measures of English language 
proficiency which are available for use by BVT projects* The Basic English Skills 
Test (see Appendi:s B) is recommended as the primary instrument for measuring 
English language proficiency » and is required for the BVT Statistical Summary 
Report* However, Individual projects may choose to use additional measures to meet 
their individual program needs* The technical data, advantages, and disadvantages 
of the various instruments presented in this paper can be used by project staff to 
begin the process of identifying these additional measures* 
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